|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.hcatalog.mapreduce.HCatOutputStorageDriver
public abstract class HCatOutputStorageDriver
The abstract class to be implemented by underlying storage drivers to enable data access from HCat through HCatOutputFormat.
Constructor Summary | |
---|---|
HCatOutputStorageDriver()
|
Method Summary | |
---|---|
void |
abortOutputCommitterJob(org.apache.hadoop.mapreduce.TaskAttemptContext context,
org.apache.hadoop.mapreduce.JobStatus.State state)
Implementation that calls the underlying output committer's abortJob, used in lieu of underlying committer's abortJob when using dynamic partitioning This should be written in a manner that is okay to call after having had multiple underlying outputcommitters write to task dirs inside it. |
void |
cleanupOutputCommitterJob(org.apache.hadoop.mapreduce.TaskAttemptContext context)
Implementation that calls the underlying output committer's cleanupJob, used in lieu of underlying committer's cleanupJob when using dynamic partitioning This should be written in a manner that is okay to call after having had multiple underlying outputcommitters write to task dirs inside it. |
abstract org.apache.hadoop.io.Writable |
convertValue(HCatRecord value)
Convert the given HCatRecord value to the actual value type. |
abstract org.apache.hadoop.io.WritableComparable<?> |
generateKey(HCatRecord value)
Generate the key for the underlying outputformat. |
abstract org.apache.hadoop.mapreduce.OutputFormat<? super org.apache.hadoop.io.WritableComparable<?>,? super org.apache.hadoop.io.Writable> |
getOutputFormat()
Returns the OutputFormat to use with this Storage Driver. |
java.lang.String |
getOutputLocation(org.apache.hadoop.mapreduce.JobContext jobContext,
java.lang.String tableLocation,
java.util.List<java.lang.String> partitionCols,
java.util.Map<java.lang.String,java.lang.String> partitionValues,
java.lang.String dynHash)
Gets the location to use for the specified partition values. |
org.apache.hadoop.fs.Path |
getWorkFilePath(org.apache.hadoop.mapreduce.TaskAttemptContext context,
java.lang.String outputLoc)
Default implementation assumes FileOutputFormat. |
void |
initialize(org.apache.hadoop.mapreduce.JobContext context,
java.util.Properties hcatProperties)
Initialize the storage driver with specified properties, default implementation does nothing. |
abstract void |
setOutputPath(org.apache.hadoop.mapreduce.JobContext jobContext,
java.lang.String location)
Set the data location for the output. |
abstract void |
setPartitionValues(org.apache.hadoop.mapreduce.JobContext jobContext,
java.util.Map<java.lang.String,java.lang.String> partitionValues)
Sets the partition key values for the partition being written. |
abstract void |
setSchema(org.apache.hadoop.mapreduce.JobContext jobContext,
HCatSchema schema)
Set the schema for the data being written out. |
void |
setupOutputCommitterJob(org.apache.hadoop.mapreduce.TaskAttemptContext context)
Implementation that calls the underlying output committer's setupJob, used in lieu of underlying committer's setupJob when using dynamic partitioning The default implementation should be overriden by underlying implementations that do not use FileOutputCommitter. |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public HCatOutputStorageDriver()
Method Detail |
---|
public void initialize(org.apache.hadoop.mapreduce.JobContext context, java.util.Properties hcatProperties) throws java.io.IOException
context
- the job context objecthcatProperties
- the properties for the storage driver
java.io.IOException
- Signals that an I/O exception has occurred.public abstract org.apache.hadoop.mapreduce.OutputFormat<? super org.apache.hadoop.io.WritableComparable<?>,? super org.apache.hadoop.io.Writable> getOutputFormat() throws java.io.IOException
java.io.IOException
- Signals that an I/O exception has occurred.public abstract void setOutputPath(org.apache.hadoop.mapreduce.JobContext jobContext, java.lang.String location) throws java.io.IOException
jobContext
- the job context objectlocation
- the data location
java.io.IOException
- Signals that an I/O exception has occurred.public abstract void setSchema(org.apache.hadoop.mapreduce.JobContext jobContext, HCatSchema schema) throws java.io.IOException
jobContext
- the job context objectschema
- the data schema
java.io.IOException
- Signals that an I/O exception has occurred.public abstract void setPartitionValues(org.apache.hadoop.mapreduce.JobContext jobContext, java.util.Map<java.lang.String,java.lang.String> partitionValues) throws java.io.IOException
jobContext
- the job context objectpartitionValues
- the partition values
java.io.IOException
- Signals that an I/O exception has occurred.public abstract org.apache.hadoop.io.WritableComparable<?> generateKey(HCatRecord value) throws java.io.IOException
value
- the value given to HCatOutputFormat
java.io.IOException
- Signals that an I/O exception has occurred.public abstract org.apache.hadoop.io.Writable convertValue(HCatRecord value) throws java.io.IOException
value
- the HCatRecord value to convert
java.io.IOException
- Signals that an I/O exception has occurred.public java.lang.String getOutputLocation(org.apache.hadoop.mapreduce.JobContext jobContext, java.lang.String tableLocation, java.util.List<java.lang.String> partitionCols, java.util.Map<java.lang.String,java.lang.String> partitionValues, java.lang.String dynHash) throws java.io.IOException
jobContext
- the job context objecttableLocation
- the location of the tablepartitionValues
- the partition valuesdynHash
- A unique hash value that represents the dynamic partitioning job used
java.io.IOException
- Signals that an I/O exception has occurred.public org.apache.hadoop.fs.Path getWorkFilePath(org.apache.hadoop.mapreduce.TaskAttemptContext context, java.lang.String outputLoc) throws java.io.IOException
java.io.IOException
public void setupOutputCommitterJob(org.apache.hadoop.mapreduce.TaskAttemptContext context) throws java.io.IOException, java.lang.InterruptedException
context
-
java.lang.InterruptedException
java.io.IOException
public void cleanupOutputCommitterJob(org.apache.hadoop.mapreduce.TaskAttemptContext context) throws java.io.IOException, java.lang.InterruptedException
context
-
java.io.IOException
java.lang.InterruptedException
public void abortOutputCommitterJob(org.apache.hadoop.mapreduce.TaskAttemptContext context, org.apache.hadoop.mapreduce.JobStatus.State state) throws java.io.IOException, java.lang.InterruptedException
context
- state
-
java.io.IOException
java.lang.InterruptedException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |