org.apache.hadoop.hive.ql.io
Class CombineHiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>

java.lang.Object
  extended by org.apache.hadoop.hive.ql.io.HiveInputFormat<K,V>
      extended by org.apache.hadoop.hive.ql.io.CombineHiveInputFormat<K,V>
All Implemented Interfaces:
org.apache.hadoop.mapred.InputFormat<K,V>, org.apache.hadoop.mapred.JobConfigurable

public class CombineHiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>
extends HiveInputFormat<K,V>

CombineHiveInputFormat is a parameterized InputFormat which looks at the path name and determine the correct InputFormat for that path name from mapredPlan.pathToPartitionInfo(). It can be used to read files with different input format in the same map-reduce job.


Nested Class Summary
static class CombineHiveInputFormat.CombineHiveInputSplit
          CombineHiveInputSplit encapsulates an InputSplit with its corresponding inputFormatClassName.
 
Nested classes/interfaces inherited from class org.apache.hadoop.hive.ql.io.HiveInputFormat
HiveInputFormat.HiveInputSplit
 
Field Summary
static org.apache.commons.logging.Log LOG
           
 
Constructor Summary
CombineHiveInputFormat()
           
 
Method Summary
protected static partitionDesc getPartitionDescFromPath(Map<String,partitionDesc> pathToPartitionInfo, org.apache.hadoop.fs.Path dir)
           
 org.apache.hadoop.mapred.RecordReader getRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.Reporter reporter)
          Create a generic Hive RecordReader than can iterate over all chunks in a CombinedFileSplit
 org.apache.hadoop.mapred.InputSplit[] getSplits(org.apache.hadoop.mapred.JobConf job, int numSplits)
          Create Hive splits based on CombineFileSplit
 
Methods inherited from class org.apache.hadoop.hive.ql.io.HiveInputFormat
configure, init, initColumnsNeeded, validateInput
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG
Constructor Detail

CombineHiveInputFormat

public CombineHiveInputFormat()
Method Detail

getSplits

public org.apache.hadoop.mapred.InputSplit[] getSplits(org.apache.hadoop.mapred.JobConf job,
                                                       int numSplits)
                                                throws IOException
Create Hive splits based on CombineFileSplit

Specified by:
getSplits in interface org.apache.hadoop.mapred.InputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>
Overrides:
getSplits in class HiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>
Throws:
IOException

getRecordReader

public org.apache.hadoop.mapred.RecordReader getRecordReader(org.apache.hadoop.mapred.InputSplit split,
                                                             org.apache.hadoop.mapred.JobConf job,
                                                             org.apache.hadoop.mapred.Reporter reporter)
                                                      throws IOException
Create a generic Hive RecordReader than can iterate over all chunks in a CombinedFileSplit

Specified by:
getRecordReader in interface org.apache.hadoop.mapred.InputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>
Overrides:
getRecordReader in class HiveInputFormat<K extends org.apache.hadoop.io.WritableComparable,V extends org.apache.hadoop.io.Writable>
Throws:
IOException

getPartitionDescFromPath

protected static partitionDesc getPartitionDescFromPath(Map<String,partitionDesc> pathToPartitionInfo,
                                                        org.apache.hadoop.fs.Path dir)
                                                 throws IOException
Throws:
IOException


Copyright © 2009 The Apache Software Foundation