org.apache.hadoop.hbase.mapreduce.hadoopbackport
Class TotalOrderPartitioner<K extends org.apache.hadoop.io.WritableComparable<?>,V>

java.lang.Object
  extended by org.apache.hadoop.mapreduce.Partitioner<K,V>
      extended by org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner<K,V>
All Implemented Interfaces:
org.apache.hadoop.conf.Configurable

public class TotalOrderPartitioner<K extends org.apache.hadoop.io.WritableComparable<?>,V>
extends org.apache.hadoop.mapreduce.Partitioner<K,V>
implements org.apache.hadoop.conf.Configurable

Partitioner effecting a total order by reading split points from an externally generated source. This is an identical copy of o.a.h.mapreduce.lib.partition.TotalOrderPartitioner from Hadoop trunk at r910774.


Field Summary
static String DEFAULT_PATH
           
static String MAX_TRIE_DEPTH
           
static String NATURAL_ORDER
           
static String PARTITIONER_PATH
           
 
Constructor Summary
TotalOrderPartitioner()
           
 
Method Summary
 org.apache.hadoop.conf.Configuration getConf()
           
 int getPartition(K key, V value, int numPartitions)
           
static String getPartitionFile(org.apache.hadoop.conf.Configuration conf)
          Get the path to the SequenceFile storing the sorted partition keyset.
 void setConf(org.apache.hadoop.conf.Configuration conf)
          Read in the partition file and build indexing data structures.
static void setPartitionFile(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.Path p)
          Set the path to the SequenceFile storing the sorted partition keyset.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

DEFAULT_PATH

public static final String DEFAULT_PATH
See Also:
Constant Field Values

PARTITIONER_PATH

public static final String PARTITIONER_PATH
See Also:
Constant Field Values

MAX_TRIE_DEPTH

public static final String MAX_TRIE_DEPTH
See Also:
Constant Field Values

NATURAL_ORDER

public static final String NATURAL_ORDER
See Also:
Constant Field Values
Constructor Detail

TotalOrderPartitioner

public TotalOrderPartitioner()
Method Detail

setConf

public void setConf(org.apache.hadoop.conf.Configuration conf)
Read in the partition file and build indexing data structures. If the keytype is BinaryComparable and total.order.partitioner.natural.order is not false, a trie of the first total.order.partitioner.max.trie.depth(2) + 1 bytes will be built. Otherwise, keys will be located using a binary search of the partition keyset using the RawComparator defined for this job. The input file must be sorted with the same comparator and contain JobContext.getNumReduceTasks() - 1 keys.

Specified by:
setConf in interface org.apache.hadoop.conf.Configurable

getConf

public org.apache.hadoop.conf.Configuration getConf()
Specified by:
getConf in interface org.apache.hadoop.conf.Configurable

getPartition

public int getPartition(K key,
                        V value,
                        int numPartitions)
Specified by:
getPartition in class org.apache.hadoop.mapreduce.Partitioner<K extends org.apache.hadoop.io.WritableComparable<?>,V>

setPartitionFile

public static void setPartitionFile(org.apache.hadoop.conf.Configuration conf,
                                    org.apache.hadoop.fs.Path p)
Set the path to the SequenceFile storing the sorted partition keyset. It must be the case that for R reduces, there are R-1 keys in the SequenceFile.


getPartitionFile

public static String getPartitionFile(org.apache.hadoop.conf.Configuration conf)
Get the path to the SequenceFile storing the sorted partition keyset.

See Also:
setPartitionFile(Configuration, Path)


Copyright © 2011 The Apache Software Foundation. All Rights Reserved.