org.apache.pig.backend.hadoop.executionengine.mapReduceLayer
Class MapReduceLauncher
java.lang.Object
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
public class MapReduceLauncher
- extends Launcher
Main class that launches pig for Map Reduce
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
SUCCEEDED_FILE_NAME
public static final String SUCCEEDED_FILE_NAME
- See Also:
- Constant Field Values
SUCCESSFUL_JOB_OUTPUT_DIR_MARKER
public static final String SUCCESSFUL_JOB_OUTPUT_DIR_MARKER
- See Also:
- Constant Field Values
MapReduceLauncher
public MapReduceLauncher()
getError
public Exception getError(FileSpec spec)
- Get the exception that caused a failure on the backend for a
store location (if any).
reset
public void reset()
- Description copied from class:
Launcher
- Resets the state after a launch
- Overrides:
reset
in class Launcher
launchPig
public PigStats launchPig(PhysicalPlan php,
String grpName,
PigContext pc)
throws PlanException,
VisitorException,
IOException,
ExecException,
JobCreationException,
Exception
- Description copied from class:
Launcher
- Method to launch pig for hadoop either for a cluster's
job tracker or for a local job runner. THe only difference
between the two is the job client. Depending on the pig context
the job client will be initialize to one of the two.
Launchers for other frameworks can overide these methods.
Given an input PhysicalPlan, it compiles it
to get a MapReduce Plan. The MapReduce plan which
has multiple MapReduce operators each one of which
has to be run as a map reduce job with dependency
information stored in the plan. It compiles the
MROperPlan into a JobControl object. Each Map Reduce
operator is converted into a Job and added to the JobControl
object. Each Job also has a set of dependent Jobs that
are created using the MROperPlan.
The JobControl object is obtained from the JobControlCompiler
Then a new thread is spawned that submits these jobs
while respecting the dependency information.
The parent thread monitors the submitted jobs' progress and
after it is complete, stops the JobControl thread.
- Specified by:
launchPig
in class Launcher
- Throws:
PlanException
VisitorException
IOException
ExecException
JobCreationException
Exception
explain
public void explain(PhysicalPlan php,
PigContext pc,
PrintStream ps,
String format,
boolean verbose)
throws PlanException,
VisitorException,
IOException
- Description copied from class:
Launcher
- Explain how a pig job will be executed on the underlying
infrastructure.
- Specified by:
explain
in class Launcher
- Parameters:
php
- PhysicalPlan to explainpc
- PigContext to use for configurationps
- PrintStream to write output on.format
- Format to write inverbose
- Amount of information to print
- Throws:
VisitorException
IOException
PlanException
compile
public MROperPlan compile(PhysicalPlan php,
PigContext pc)
throws PlanException,
IOException,
VisitorException
- Throws:
PlanException
IOException
VisitorException
Copyright © 2007-2012 The Apache Software Foundation