|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.pig.backend.hadoop.executionengine.util.MapRedUtil
public class MapRedUtil
A class of utility static methods to be used in the hadoop map reduce backend
Constructor Summary | |
---|---|
MapRedUtil()
|
Method Summary | ||
---|---|---|
static StoreConfig |
getStoreConfig(org.apache.hadoop.mapred.JobConf conf)
This method is to be called from an OutputFormat.getRecordWriter(FileSystem ignored, JobConf job,
String name, Progressable progress)
method to obtain a reference to the StoreConfig object. |
|
static StoreFunc |
getStoreFunc(org.apache.hadoop.mapred.JobConf conf)
This method is to be called from an OutputFormat.getRecordWriter(FileSystem ignored, JobConf job,
String name, Progressable progress)
method to obtain a reference to the StoreFunc object to be used by
that OutputFormat to perform the write() operation |
|
static
|
loadPartitionFile(String keyDistFile,
Integer[] totalReducers,
org.apache.hadoop.mapred.JobConf job,
byte keyType)
Loads the key distribution sampler file |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Constructor Detail |
---|
public MapRedUtil()
Method Detail |
---|
public static StoreFunc getStoreFunc(org.apache.hadoop.mapred.JobConf conf) throws ExecException
OutputFormat.getRecordWriter(FileSystem ignored, JobConf job,
String name, Progressable progress)
method to obtain a reference to the StoreFunc
object to be used by
that OutputFormat to perform the write() operation
conf
- the JobConf object
ExecException
public static StoreConfig getStoreConfig(org.apache.hadoop.mapred.JobConf conf) throws IOException
OutputFormat.getRecordWriter(FileSystem ignored, JobConf job,
String name, Progressable progress)
method to obtain a reference to the StoreConfig
object. The StoreConfig
object will contain metadata information like schema and location to be used by
that OutputFormat to perform the write() operation
conf
- the JobConf object
IOException
public static <E> Map<E,Pair<Integer,Integer>> loadPartitionFile(String keyDistFile, Integer[] totalReducers, org.apache.hadoop.mapred.JobConf job, byte keyType) throws IOException
keyDistFile
- the name for the distribution filetotalReducers
- gets set to the total number of reducers as found in the dist filejob
- Ref to a jobCong objectkeyType
- Type of the key to be stored in the return map. It currently treats Tuple as a special case.
IOException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |