|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
public class JobControlCompiler
This is compiler class that takes an MROperPlan and converts it into a JobControl object with the relevant dependency info maintained. The JobControl Object is made up of Jobs each of which has a JobConf. The MapReduceOper corresponds to a Job and the getJobCong method returns the JobConf that is configured as per the MapReduceOper
A few words on how comparators are chosen. In almost all cases we use raw comparators (the one exception being when the user provides a comparison function for order by). For order by queries the PigTYPERawComparator functions are used, where TYPE is Int, Long, etc. These comparators are null aware and asc/desc aware. The first byte of each of the NullableTYPEWritable classes contains info on whether the value is null. Asc/desc is written as an array into the JobConf with the key pig.sortOrder so that it can be read by each of the comparators as part of their setConf call.
For non-order by queries, PigTYPEWritableComparator classes are used. These are all just type specific instances of WritableComparator.
Nested Class Summary | |
---|---|
static class |
JobControlCompiler.PigBagWritableComparator
|
static class |
JobControlCompiler.PigCharArrayWritableComparator
|
static class |
JobControlCompiler.PigDBAWritableComparator
|
static class |
JobControlCompiler.PigDoubleWritableComparator
|
static class |
JobControlCompiler.PigFloatWritableComparator
|
static class |
JobControlCompiler.PigIntWritableComparator
|
static class |
JobControlCompiler.PigLongWritableComparator
|
static class |
JobControlCompiler.PigSecondaryKeyGroupComparator
|
static class |
JobControlCompiler.PigTupleWritableComparator
|
static class |
JobControlCompiler.PigWritableComparator
|
Field Summary | |
---|---|
static String |
END_OF_INP_IN_MAP
|
static String |
LOG_DIR
|
static String |
PIG_STORE_CONFIG
|
Constructor Summary | |
---|---|
JobControlCompiler(PigContext pigContext,
org.apache.hadoop.conf.Configuration conf)
|
Method Summary | |
---|---|
org.apache.hadoop.mapred.jobcontrol.JobControl |
compile(MROperPlan plan,
String grpName)
Compiles all jobs that have no dependencies removes them from the plan and returns. |
List<POStore> |
getStores(org.apache.hadoop.mapred.jobcontrol.Job job)
Returns all store locations of a previously compiled job |
void |
moveResults(List<org.apache.hadoop.mapred.jobcontrol.Job> completedJobs)
Moves all the results of a collection of MR jobs to the final output directory. |
void |
reset()
Resets the state |
int |
updateMROpPlan(List<org.apache.hadoop.mapred.jobcontrol.Job> completeFailedJobs)
|
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final String PIG_STORE_CONFIG
public static final String LOG_DIR
public static final String END_OF_INP_IN_MAP
Constructor Detail |
---|
public JobControlCompiler(PigContext pigContext, org.apache.hadoop.conf.Configuration conf) throws IOException
IOException
Method Detail |
---|
public List<POStore> getStores(org.apache.hadoop.mapred.jobcontrol.Job job)
public void reset()
public void moveResults(List<org.apache.hadoop.mapred.jobcontrol.Job> completedJobs) throws IOException
IOException
public org.apache.hadoop.mapred.jobcontrol.JobControl compile(MROperPlan plan, String grpName) throws JobCreationException
plan
- - The MROperPlan to be compiledgrpName
- - The name given to the JobControl
JobCreationException
public int updateMROpPlan(List<org.apache.hadoop.mapred.jobcontrol.Job> completeFailedJobs)
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |