org.apache.hadoop.hive.ql.exec
Class FileSinkOperator

java.lang.Object
  extended by org.apache.hadoop.hive.ql.exec.Operator<T>
      extended by org.apache.hadoop.hive.ql.exec.TerminalOperator<fileSinkDesc>
          extended by org.apache.hadoop.hive.ql.exec.FileSinkOperator
All Implemented Interfaces:
Serializable, Node

public class FileSinkOperator
extends TerminalOperator<fileSinkDesc>
implements Serializable

File Sink operator implementation

See Also:
Serialized Form

Nested Class Summary
static interface FileSinkOperator.RecordWriter
           
static class FileSinkOperator.TableIdEnum
           
 
Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.exec.Operator
Operator.OperatorFunc, Operator.ProgressCounter, Operator.State
 
Field Summary
protected  boolean autoDelete
           
protected  org.apache.hadoop.io.BytesWritable commonKey
           
protected  org.apache.hadoop.fs.Path finalPath
           
protected  org.apache.hadoop.fs.FileSystem fs
           
protected  org.apache.hadoop.fs.Path outPath
           
protected  FileSinkOperator.RecordWriter outWriter
           
protected  Serializer serializer
           
protected  FileSinkOperator.TableIdEnum tabIdEnum
           
 
Fields inherited from class org.apache.hadoop.hive.ql.exec.Operator
alias, beginTime, childOperators, childOperatorsArray, childOperatorsTag, colExprMap, conf, counterNames, counterNameToEnum, counters, done, fatalErrorCntr, id, inputObjInspectors, inputRows, LOG, numInputRowsCntr, numOutputRowsCntr, operatorId, out, outputObjInspector, outputRows, parentOperators, reporter, state, statsMap, timeTakenCntr, totalTime
 
Constructor Summary
FileSinkOperator()
           
 
Method Summary
 void closeOp(boolean abort)
          Operator specific close routine.
 String getName()
          Implements the getName function for the Node Interface.
static FileSinkOperator.RecordWriter getRecordWriter(org.apache.hadoop.mapred.JobConf jc, HiveOutputFormat<?,?> hiveOutputFormat, Class<? extends org.apache.hadoop.io.Writable> valueClass, boolean isCompressed, Properties tableProp, org.apache.hadoop.fs.Path outPath)
           
 int getType()
          Should be overridden to return the type of the specific operator among the types in OperatorType
protected  void initializeOp(org.apache.hadoop.conf.Configuration hconf)
          Operator specific initialization.
 void jobClose(org.apache.hadoop.conf.Configuration hconf, boolean success)
          Unlike other operator interfaces which are called from map or reduce task, jobClose is called from the jobclient side once the job has completed
 void processOp(Object row, int tag)
          Process the row.
 
Methods inherited from class org.apache.hadoop.hive.ql.exec.Operator
areAllParentsInitialized, assignCounterNameToEnum, checkFatalErrors, close, dump, dump, endGroup, fatalErrorMessage, forward, getChildOperators, getChildren, getColumnExprMap, getConf, getCounterNames, getCounterNameToEnum, getCounters, getDone, getIdentifier, getOperatorId, getParentOperators, getSchema, getStats, incrCounter, initEvaluators, initEvaluatorsAndReturnStruct, initialize, initializeChildren, initializeCounters, initOperatorId, logStats, preorderMap, process, removeChild, replaceChild, replaceParent, resetId, resetLastEnumUsed, resetStats, setAlias, setChildOperators, setColumnExprMap, setConf, setCounterNames, setCounterNameToEnum, setDone, setId, setOperatorId, setOutputCollector, setParentOperators, setReporter, setSchema, startGroup, updateCounters
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

outWriter

protected transient FileSinkOperator.RecordWriter outWriter

fs

protected transient org.apache.hadoop.fs.FileSystem fs

outPath

protected transient org.apache.hadoop.fs.Path outPath

finalPath

protected transient org.apache.hadoop.fs.Path finalPath

serializer

protected transient Serializer serializer

commonKey

protected transient org.apache.hadoop.io.BytesWritable commonKey

tabIdEnum

protected transient FileSinkOperator.TableIdEnum tabIdEnum

autoDelete

protected transient boolean autoDelete
Constructor Detail

FileSinkOperator

public FileSinkOperator()
Method Detail

initializeOp

protected void initializeOp(org.apache.hadoop.conf.Configuration hconf)
                     throws HiveException
Description copied from class: Operator
Operator specific initialization.

Overrides:
initializeOp in class Operator<fileSinkDesc>
Throws:
HiveException

getRecordWriter

public static FileSinkOperator.RecordWriter getRecordWriter(org.apache.hadoop.mapred.JobConf jc,
                                                            HiveOutputFormat<?,?> hiveOutputFormat,
                                                            Class<? extends org.apache.hadoop.io.Writable> valueClass,
                                                            boolean isCompressed,
                                                            Properties tableProp,
                                                            org.apache.hadoop.fs.Path outPath)
                                                     throws IOException,
                                                            HiveException
Throws:
IOException
HiveException

processOp

public void processOp(Object row,
                      int tag)
               throws HiveException
Description copied from class: Operator
Process the row.

Specified by:
processOp in class Operator<fileSinkDesc>
Parameters:
row - The object representing the row.
tag - The tag of the row usually means which parent this row comes from. Rows with the same tag should have exactly the same rowInspector all the time.
Throws:
HiveException

closeOp

public void closeOp(boolean abort)
             throws HiveException
Description copied from class: Operator
Operator specific close routine. Operators which inherents this class should overwrite this funtion for their specific cleanup routine.

Overrides:
closeOp in class Operator<fileSinkDesc>
Throws:
HiveException

getName

public String getName()
Description copied from class: Operator
Implements the getName function for the Node Interface.

Specified by:
getName in interface Node
Overrides:
getName in class Operator<fileSinkDesc>
Returns:
the name of the operator

jobClose

public void jobClose(org.apache.hadoop.conf.Configuration hconf,
                     boolean success)
              throws HiveException
Description copied from class: Operator
Unlike other operator interfaces which are called from map or reduce task, jobClose is called from the jobclient side once the job has completed

Overrides:
jobClose in class Operator<fileSinkDesc>
Parameters:
hconf - Configuration with with which job was submitted
success - whether the job was completed successfully or not
Throws:
HiveException

getType

public int getType()
Description copied from class: Operator
Should be overridden to return the type of the specific operator among the types in OperatorType

Overrides:
getType in class Operator<fileSinkDesc>
Returns:
OperatorType.* or -1 if not overridden


Copyright © 2009 The Apache Software Foundation