|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
public class JobControlCompiler
This is compiler class that takes an MROperPlan and converts it into a JobControl object with the relevant dependency info maintained. The JobControl Object is made up of Jobs each of which has a JobConf. The MapReduceOper corresponds to a Job and the getJobCong method returns the JobConf that is configured as per the MapReduceOper
A few words on how comparators are chosen. In almost all cases we use raw comparators (the one exception being when the user provides a comparison function for order by). For order by queries the PigTYPERawComparator functions are used, where TYPE is Int, Long, etc. These comparators are null aware and asc/desc aware. The first byte of each of the NullableTYPEWritable classes contains info on whether the value is null. Asc/desc is written as an array into the JobConf with the key pig.sortOrder so that it can be read by each of the comparators as part of their setConf call.
For non-order by queries, PigTYPEWritableComparator classes are used. These are all just type specific instances of WritableComparator.
Field Summary | |
---|---|
static String |
END_OF_INP_IN_MAP
|
static String |
LOG_DIR
|
static String |
PIG_MAP_STORES
We will serialize the POStore(s) present in map and reduce in lists in the Hadoop Conf. |
static String |
PIG_REDUCE_STORES
|
Constructor Summary | |
---|---|
JobControlCompiler(PigContext pigContext,
org.apache.hadoop.conf.Configuration conf)
|
Method Summary | |
---|---|
org.apache.hadoop.mapred.jobcontrol.JobControl |
compile(MROperPlan plan,
String grpName)
Compiles all jobs that have no dependencies removes them from the plan and returns. |
Map<org.apache.hadoop.mapred.jobcontrol.Job,MapReduceOper> |
getJobMroMap()
Gets the map of Job and the MR Operator |
List<POStore> |
getStores(org.apache.hadoop.mapred.jobcontrol.Job job)
Returns all store locations of a previously compiled job |
void |
moveResults(List<org.apache.hadoop.mapred.jobcontrol.Job> completedJobs)
Moves all the results of a collection of MR jobs to the final output directory. |
void |
reset()
Resets the state |
int |
updateMROpPlan(List<org.apache.hadoop.mapred.jobcontrol.Job> completeFailedJobs)
|
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final String LOG_DIR
public static final String END_OF_INP_IN_MAP
public static final String PIG_MAP_STORES
public static final String PIG_REDUCE_STORES
Constructor Detail |
---|
public JobControlCompiler(PigContext pigContext, org.apache.hadoop.conf.Configuration conf) throws IOException
IOException
Method Detail |
---|
public List<POStore> getStores(org.apache.hadoop.mapred.jobcontrol.Job job)
public void reset()
public Map<org.apache.hadoop.mapred.jobcontrol.Job,MapReduceOper> getJobMroMap()
public void moveResults(List<org.apache.hadoop.mapred.jobcontrol.Job> completedJobs) throws IOException
IOException
public org.apache.hadoop.mapred.jobcontrol.JobControl compile(MROperPlan plan, String grpName) throws JobCreationException
plan
- - The MROperPlan to be compiledgrpName
- - The name given to the JobControl
JobCreationException
public int updateMROpPlan(List<org.apache.hadoop.mapred.jobcontrol.Job> completeFailedJobs)
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |