org.apache.pig.backend.hadoop.executionengine.util
Class MapRedUtil
java.lang.Object
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil
public class MapRedUtil
- extends Object
A class of utility static methods to be used in the hadoop map reduce backend
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
FILE_SYSTEM_NAME
public static final String FILE_SYSTEM_NAME
- See Also:
- Constant Field Values
MapRedUtil
public MapRedUtil()
loadPartitionFileFromLocalCache
public static <E> Map<E,Pair<Integer,Integer>> loadPartitionFileFromLocalCache(String keyDistFile,
Integer[] totalReducers,
byte keyType,
org.apache.hadoop.conf.Configuration mapConf)
throws IOException
- Loads the key distribution sampler file
- Parameters:
keyDistFile
- the name for the distribution filetotalReducers
- gets set to the total number of reducers as found in the dist filekeyType
- Type of the key to be stored in the return map. It currently treats Tuple as a special case.
- Throws:
IOException
setupUDFContext
public static void setupUDFContext(org.apache.hadoop.conf.Configuration job)
throws IOException
- Throws:
IOException
checkLeafIsStore
public static FileSpec checkLeafIsStore(PhysicalPlan plan,
PigContext pigContext)
throws ExecException
- Throws:
ExecException
getAllFileRecursively
public static List<org.apache.hadoop.fs.FileStatus> getAllFileRecursively(List<org.apache.hadoop.fs.FileStatus> files,
org.apache.hadoop.conf.Configuration conf)
throws IOException
- Get all files recursively from the given list of files
- Parameters:
files
- a list of FileStatusconf
- the configuration object
- Returns:
- the list of fileStatus that contains all the files in the given
list and, recursively, all the files inside the directories in
the given list
- Throws:
IOException
getCombinePigSplits
public static List<List<org.apache.hadoop.mapreduce.InputSplit>> getCombinePigSplits(List<org.apache.hadoop.mapreduce.InputSplit> oneInputSplits,
long maxCombinedSplitSize,
org.apache.hadoop.conf.Configuration conf)
throws IOException,
InterruptedException
- Throws:
IOException
InterruptedException
inputSplitToString
public String inputSplitToString(org.apache.hadoop.mapreduce.InputSplit[] splits)
throws IOException,
InterruptedException
- Throws:
IOException
InterruptedException
Copyright © 2007-2012 The Apache Software Foundation