org.apache.hadoop.hive.ql.exec
Class ExecDriver

java.lang.Object
  extended by org.apache.hadoop.hive.ql.exec.Task<mapredWork>
      extended by org.apache.hadoop.hive.ql.exec.ExecDriver
All Implemented Interfaces:
Serializable

public class ExecDriver
extends Task<mapredWork>
implements Serializable

See Also:
Serialized Form

Nested Class Summary
static class ExecDriver.ExecDriverTaskHandle
          This class contains the state of the running task Going forward, we will return this handle from execute and Driver can split execute into start, monitorProgess and postProcess
 
Field Summary
protected  org.apache.hadoop.mapred.JobConf job
           
protected  int mapProgress
           
static Random randGen
           
protected  int reduceProgress
           
static Map<String,String> runningJobKillURIs
          A list of the currently running jobs spawned in this Hive instance that is used to kill all running jobs in the event of an unexpected shutdown - i.e., the JVM shuts down while there are still jobs running.
protected  boolean success
           
 
Fields inherited from class org.apache.hadoop.hive.ql.exec.Task
childTasks, conf, console, db, id, initialized, isdone, LOG, parentTasks, queryPlan, queued, started, taskCounters, taskHandle, work
 
Constructor Summary
ExecDriver()
          Constructor when invoked from QL
ExecDriver(mapredWork plan, org.apache.hadoop.mapred.JobConf job, boolean isSilent)
          Constructor/Initialization for invocation as independent utility
 
Method Summary
protected  boolean checkFatalErrors(TaskHandle t, StringBuffer errMsg)
          Fatal errors are those errors that cannot be recovered by retries.
 int estimateNumberOfReducers(HiveConf hive, org.apache.hadoop.mapred.JobConf job, mapredWork work)
          Estimate the number of reducers needed for this job, based on job input, and configuration parameters.
 int execute()
          Execute a query plan using Hadoop
static String generateCmdLine(HiveConf hconf)
          Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver
static String getResourceFiles(org.apache.hadoop.conf.Configuration conf, SessionState.ResourceType t)
           
 long getTotalInputFileSize(org.apache.hadoop.mapred.JobConf job, mapredWork work)
          Calculate the total size of input files.
 int getType()
          Should be overridden to return the type of the specific task among the types in TaskType
 boolean hasReduce()
           
 void initialize(HiveConf conf, QueryPlan queryPlan)
          Initialization when invoked from QL
 boolean isMapRedTask()
           
 void jobInfo(org.apache.hadoop.mapred.RunningJob rj)
          from StreamJob.java
static void main(String[] args)
           
 boolean mapDone()
           
 boolean mapStarted()
           
 void progress(TaskHandle taskHandle)
          Update the progress of the task within taskHandle and also dump the progress information to the history file
 boolean reduceDone()
           
 boolean reduceStarted()
           
protected  void setNumberOfReducers()
          Set the number of reducers for the mapred work.
 void updateCounters(TaskHandle t)
          update counters relevant to this task
 
Methods inherited from class org.apache.hadoop.hive.ql.exec.Task
addDependentTask, done, executeTask, fetch, getChildTasks, getCounters, getId, getInitialized, getParentTasks, getQueued, getWork, isRunnable, removeDependentTask, setChildTasks, setDone, setId, setInitialized, setParentTasks, setQueued, setStarted, setWork, started
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

job

protected transient org.apache.hadoop.mapred.JobConf job

mapProgress

protected transient int mapProgress

reduceProgress

protected transient int reduceProgress

success

protected transient boolean success

randGen

public static Random randGen

runningJobKillURIs

public static Map<String,String> runningJobKillURIs
A list of the currently running jobs spawned in this Hive instance that is used to kill all running jobs in the event of an unexpected shutdown - i.e., the JVM shuts down while there are still jobs running.

Constructor Detail

ExecDriver

public ExecDriver()
Constructor when invoked from QL


ExecDriver

public ExecDriver(mapredWork plan,
                  org.apache.hadoop.mapred.JobConf job,
                  boolean isSilent)
           throws HiveException
Constructor/Initialization for invocation as independent utility

Throws:
HiveException
Method Detail

getResourceFiles

public static String getResourceFiles(org.apache.hadoop.conf.Configuration conf,
                                      SessionState.ResourceType t)

initialize

public void initialize(HiveConf conf,
                       QueryPlan queryPlan)
Initialization when invoked from QL

Overrides:
initialize in class Task<mapredWork>

jobInfo

public void jobInfo(org.apache.hadoop.mapred.RunningJob rj)
from StreamJob.java


checkFatalErrors

protected boolean checkFatalErrors(TaskHandle t,
                                   StringBuffer errMsg)
Fatal errors are those errors that cannot be recovered by retries. These are application dependent. Examples of fatal errors include: - the small table in the map-side joins is too large to be feasible to be handled by one mapper. The job should fail and the user should be warned to use regular joins rather than map-side joins. Fatal errors are indicated by counters that are set at execution time. If the counter is non-zero, a fatal error occurred. The value of the counter indicates the error type.

Returns:
true if fatal errors happened during job execution, false otherwise.

progress

public void progress(TaskHandle taskHandle)
              throws IOException
Description copied from class: Task
Update the progress of the task within taskHandle and also dump the progress information to the history file

Overrides:
progress in class Task<mapredWork>
Parameters:
taskHandle - task handle returned by execute
Throws:
IOException

estimateNumberOfReducers

public int estimateNumberOfReducers(HiveConf hive,
                                    org.apache.hadoop.mapred.JobConf job,
                                    mapredWork work)
                             throws IOException
Estimate the number of reducers needed for this job, based on job input, and configuration parameters.

Returns:
the number of reducers.
Throws:
IOException

setNumberOfReducers

protected void setNumberOfReducers()
                            throws IOException
Set the number of reducers for the mapred work.

Throws:
IOException

getTotalInputFileSize

public long getTotalInputFileSize(org.apache.hadoop.mapred.JobConf job,
                                  mapredWork work)
                           throws IOException
Calculate the total size of input files.

Parameters:
job - the hadoop job conf.
Returns:
the total size in bytes.
Throws:
IOException

updateCounters

public void updateCounters(TaskHandle t)
                    throws IOException
update counters relevant to this task

Overrides:
updateCounters in class Task<mapredWork>
Throws:
IOException

mapStarted

public boolean mapStarted()

reduceStarted

public boolean reduceStarted()

mapDone

public boolean mapDone()

reduceDone

public boolean reduceDone()

execute

public int execute()
Execute a query plan using Hadoop

Specified by:
execute in class Task<mapredWork>
Returns:
status of executing the task

main

public static void main(String[] args)
                 throws IOException,
                        HiveException
Throws:
IOException
HiveException

generateCmdLine

public static String generateCmdLine(HiveConf hconf)
Given a Hive Configuration object - generate a command line fragment for passing such configuration information to ExecDriver


isMapRedTask

public boolean isMapRedTask()
Overrides:
isMapRedTask in class Task<mapredWork>

hasReduce

public boolean hasReduce()
Overrides:
hasReduce in class Task<mapredWork>

getType

public int getType()
Description copied from class: Task
Should be overridden to return the type of the specific task among the types in TaskType

Overrides:
getType in class Task<mapredWork>
Returns:
TaskTypeType.* or -1 if not overridden


Copyright © 2009 The Apache Software Foundation