org.apache.hadoop.hive.ql.exec
Class ExecDriver

java.lang.Object
  extended by org.apache.hadoop.hive.ql.exec.Task<MapredWork>
      extended by org.apache.hadoop.hive.ql.exec.ExecDriver
All Implemented Interfaces:
Serializable, Node

public class ExecDriver
extends Task<MapredWork>
implements Serializable

ExecDriver.

See Also:
Serialized Form

Nested Class Summary
static class ExecDriver.ExecDriverTaskHandle
          This class contains the state of the running task Going forward, we will return this handle from execute and Driver can split execute into start, monitorProgess and postProcess.
 
Field Summary
protected  org.apache.hadoop.mapred.JobConf job
           
protected  int mapProgress
           
static Random randGen
           
protected  int reduceProgress
           
static Map<String,String> runningJobKillURIs
          A list of the currently running jobs spawned in this Hive instance that is used to kill all running jobs in the event of an unexpected shutdown - i.e., the JVM shuts down while there are still jobs running.
protected  boolean success
           
 
Fields inherited from class org.apache.hadoop.hive.ql.exec.Task
childTasks, conf, console, db, driverContext, id, initialized, isdone, LOG, parentTasks, queryPlan, queued, started, taskCounters, taskHandle, work
 
Constructor Summary
ExecDriver()
          Constructor when invoked from QL.
ExecDriver(MapredWork plan, org.apache.hadoop.mapred.JobConf job, boolean isSilent)
          Constructor/Initialization for invocation as independent utility.
 
Method Summary
protected  boolean checkFatalErrors(TaskHandle t, StringBuilder errMsg)
          Fatal errors are those errors that cannot be recovered by retries.
 int estimateNumberOfReducers(HiveConf hive, org.apache.hadoop.mapred.JobConf job, MapredWork work)
          Estimate the number of reducers needed for this job, based on job input, and configuration parameters.
 int execute(DriverContext driverContext)
          Execute a query plan using Hadoop.
static String generateCmdLine(HiveConf hconf)
          Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver.
static String getJobEndMsg(String jobId)
          this msg pattern is used to track when a job is successfully done.
static String getJobStartMsg(String jobId)
          This msg pattern is used to track when a job is started.
 String getName()
          Gets the name of the node.
static String getResourceFiles(org.apache.hadoop.conf.Configuration conf, SessionState.ResourceType t)
           
 long getTotalInputFileSize(org.apache.hadoop.mapred.JobConf job, MapredWork work)
          Calculate the total size of input files.
 int getType()
          Should be overridden to return the type of the specific task among the types in TaskType.
 boolean hasReduce()
           
 void initialize(HiveConf conf, QueryPlan queryPlan, DriverContext driverContext)
          Initialization when invoked from QL.
 boolean isMapRedTask()
           
 void jobInfo(org.apache.hadoop.mapred.RunningJob rj)
          from StreamJob.java.
static void main(String[] args)
           
 boolean mapDone()
           
 boolean mapStarted()
           
 void progress(TaskHandle taskHandle)
          Update the progress of the task within taskHandle and also dump the progress information to the history file.
 boolean reduceDone()
           
 boolean reduceStarted()
           
protected  void setNumberOfReducers()
          Set the number of reducers for the mapred work.
 void updateCounters(TaskHandle t)
          Update counters relevant to this task.
 
Methods inherited from class org.apache.hadoop.hive.ql.exec.Task
addDependentTask, done, executeTask, fetch, getChildren, getChildTasks, getCounters, getDependentTasks, getId, getInitialized, getParentTasks, getQueued, getWork, isRunnable, removeDependentTask, setChildTasks, setDone, setId, setInitialized, setParentTasks, setQueued, setStarted, setWork, started
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

job

protected transient org.apache.hadoop.mapred.JobConf job

mapProgress

protected transient int mapProgress

reduceProgress

protected transient int reduceProgress

success

protected transient boolean success

randGen

public static Random randGen

runningJobKillURIs

public static Map<String,String> runningJobKillURIs
A list of the currently running jobs spawned in this Hive instance that is used to kill all running jobs in the event of an unexpected shutdown - i.e., the JVM shuts down while there are still jobs running.

Constructor Detail

ExecDriver

public ExecDriver()
Constructor when invoked from QL.


ExecDriver

public ExecDriver(MapredWork plan,
                  org.apache.hadoop.mapred.JobConf job,
                  boolean isSilent)
           throws HiveException
Constructor/Initialization for invocation as independent utility.

Throws:
HiveException
Method Detail

getResourceFiles

public static String getResourceFiles(org.apache.hadoop.conf.Configuration conf,
                                      SessionState.ResourceType t)

initialize

public void initialize(HiveConf conf,
                       QueryPlan queryPlan,
                       DriverContext driverContext)
Initialization when invoked from QL.

Overrides:
initialize in class Task<MapredWork>

jobInfo

public void jobInfo(org.apache.hadoop.mapred.RunningJob rj)
from StreamJob.java.


checkFatalErrors

protected boolean checkFatalErrors(TaskHandle t,
                                   StringBuilder errMsg)
Fatal errors are those errors that cannot be recovered by retries. These are application dependent. Examples of fatal errors include: - the small table in the map-side joins is too large to be feasible to be handled by one mapper. The job should fail and the user should be warned to use regular joins rather than map-side joins. Fatal errors are indicated by counters that are set at execution time. If the counter is non-zero, a fatal error occurred. The value of the counter indicates the error type.

Returns:
true if fatal errors happened during job execution, false otherwise.

progress

public void progress(TaskHandle taskHandle)
              throws IOException
Description copied from class: Task
Update the progress of the task within taskHandle and also dump the progress information to the history file.

Overrides:
progress in class Task<MapredWork>
Parameters:
taskHandle - task handle returned by execute
Throws:
IOException

estimateNumberOfReducers

public int estimateNumberOfReducers(HiveConf hive,
                                    org.apache.hadoop.mapred.JobConf job,
                                    MapredWork work)
                             throws IOException
Estimate the number of reducers needed for this job, based on job input, and configuration parameters.

Returns:
the number of reducers.
Throws:
IOException

setNumberOfReducers

protected void setNumberOfReducers()
                            throws IOException
Set the number of reducers for the mapred work.

Throws:
IOException

getTotalInputFileSize

public long getTotalInputFileSize(org.apache.hadoop.mapred.JobConf job,
                                  MapredWork work)
                           throws IOException
Calculate the total size of input files.

Parameters:
job - the hadoop job conf.
Returns:
the total size in bytes.
Throws:
IOException

updateCounters

public void updateCounters(TaskHandle t)
                    throws IOException
Update counters relevant to this task.

Overrides:
updateCounters in class Task<MapredWork>
Throws:
IOException

mapStarted

public boolean mapStarted()

reduceStarted

public boolean reduceStarted()

mapDone

public boolean mapDone()

reduceDone

public boolean reduceDone()

execute

public int execute(DriverContext driverContext)
Execute a query plan using Hadoop.

Specified by:
execute in class Task<MapredWork>
Returns:
status of executing the task

getJobStartMsg

public static String getJobStartMsg(String jobId)
This msg pattern is used to track when a job is started.

Parameters:
jobId -
Returns:

getJobEndMsg

public static String getJobEndMsg(String jobId)
this msg pattern is used to track when a job is successfully done.

Parameters:
jobId -
Returns:

main

public static void main(String[] args)
                 throws IOException,
                        HiveException
Throws:
IOException
HiveException

generateCmdLine

public static String generateCmdLine(HiveConf hconf)
Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver.


isMapRedTask

public boolean isMapRedTask()
Overrides:
isMapRedTask in class Task<MapredWork>

hasReduce

public boolean hasReduce()
Overrides:
hasReduce in class Task<MapredWork>

getType

public int getType()
Description copied from class: Task
Should be overridden to return the type of the specific task among the types in TaskType.

Overrides:
getType in class Task<MapredWork>
Returns:
TaskTypeType.* or -1 if not overridden

getName

public String getName()
Description copied from interface: Node
Gets the name of the node. This is used in the rule dispatchers.

Specified by:
getName in interface Node
Returns:
String


Copyright © 2010 The Apache Software Foundation