org.apache.hadoop.hive.ql.optimizer
Class GenMapRedUtils

java.lang.Object
  extended by org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils

public class GenMapRedUtils
extends Object

General utility common functions for the Processor to convert operator into map-reduce tasks


Constructor Summary
GenMapRedUtils()
           
 
Method Summary
static mapredWork getMapRedWork()
          create a new plan and return
static void initMapJoinPlan(Operator<? extends Serializable> op, GenMRProcContext opProcCtx, boolean readInputMapJoin, boolean readInputUnion, boolean setReducer, int pos)
          Initialize the current plan by adding it to root tasks
static void initPlan(ReduceSinkOperator op, GenMRProcContext opProcCtx)
          Initialize the current plan by adding it to root tasks
static void initUnionPlan(GenMRProcContext opProcCtx, Task<? extends Serializable> currTask, boolean local)
           
static void initUnionPlan(ReduceSinkOperator op, GenMRProcContext opProcCtx)
          Initialize the current union plan.
static void joinPlan(Operator<? extends Serializable> op, Task<? extends Serializable> oldTask, Task<? extends Serializable> task, GenMRProcContext opProcCtx, int pos, boolean split, boolean readMapJoinData, boolean readUnionData)
          Merge the current task with the task for the current reducer
static void mergeMapJoinUnion(UnionOperator union, GenMRProcContext ctx, int pos)
           
static void setKeyAndValueDesc(mapredWork plan, Operator<? extends Serializable> topOp)
          set key and value descriptor
static void setTaskPlan(String alias_id, Operator<? extends Serializable> topOp, mapredWork plan, boolean local, GenMRProcContext opProcCtx)
          set the current task in the mapredWork
static void setTaskPlan(String path, String alias, Operator<? extends Serializable> topOp, mapredWork plan, boolean local, tableDesc tt_desc)
          set the current task in the mapredWork
static void splitPlan(ReduceSinkOperator op, GenMRProcContext opProcCtx)
          Split the current plan by creating a temporary destination
static void splitTasks(Operator<? extends Serializable> op, Task<? extends Serializable> parentTask, Task<? extends Serializable> childTask, GenMRProcContext opProcCtx, boolean setReducer, boolean local, int posn)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

GenMapRedUtils

public GenMapRedUtils()
Method Detail

initPlan

public static void initPlan(ReduceSinkOperator op,
                            GenMRProcContext opProcCtx)
                     throws SemanticException
Initialize the current plan by adding it to root tasks

Parameters:
op - the reduce sink operator encountered
opProcCtx - processing context
Throws:
SemanticException

initMapJoinPlan

public static void initMapJoinPlan(Operator<? extends Serializable> op,
                                   GenMRProcContext opProcCtx,
                                   boolean readInputMapJoin,
                                   boolean readInputUnion,
                                   boolean setReducer,
                                   int pos)
                            throws SemanticException
Initialize the current plan by adding it to root tasks

Parameters:
op - the map join operator encountered
opProcCtx - processing context
pos - position of the parent
Throws:
SemanticException

initUnionPlan

public static void initUnionPlan(ReduceSinkOperator op,
                                 GenMRProcContext opProcCtx)
                          throws SemanticException
Initialize the current union plan.

Parameters:
op - the reduce sink operator encountered
opProcCtx - processing context
Throws:
SemanticException

initUnionPlan

public static void initUnionPlan(GenMRProcContext opProcCtx,
                                 Task<? extends Serializable> currTask,
                                 boolean local)

joinPlan

public static void joinPlan(Operator<? extends Serializable> op,
                            Task<? extends Serializable> oldTask,
                            Task<? extends Serializable> task,
                            GenMRProcContext opProcCtx,
                            int pos,
                            boolean split,
                            boolean readMapJoinData,
                            boolean readUnionData)
                     throws SemanticException
Merge the current task with the task for the current reducer

Parameters:
op - operator being processed
oldTask - the old task for the current reducer
task - the current task for the current reducer
opProcCtx - processing context
pos - position of the parent in the stack
Throws:
SemanticException

splitPlan

public static void splitPlan(ReduceSinkOperator op,
                             GenMRProcContext opProcCtx)
                      throws SemanticException
Split the current plan by creating a temporary destination

Parameters:
op - the reduce sink operator encountered
opProcCtx - processing context
Throws:
SemanticException

setTaskPlan

public static void setTaskPlan(String alias_id,
                               Operator<? extends Serializable> topOp,
                               mapredWork plan,
                               boolean local,
                               GenMRProcContext opProcCtx)
                        throws SemanticException
set the current task in the mapredWork

Parameters:
alias_id - current alias
topOp - the top operator of the stack
plan - current plan
local - whether you need to add to map-reduce or local work
opProcCtx - processing context
Throws:
SemanticException

setTaskPlan

public static void setTaskPlan(String path,
                               String alias,
                               Operator<? extends Serializable> topOp,
                               mapredWork plan,
                               boolean local,
                               tableDesc tt_desc)
                        throws SemanticException
set the current task in the mapredWork

Parameters:
alias - current alias
topOp - the top operator of the stack
plan - current plan
local - whether you need to add to map-reduce or local work
tt_desc - table descriptor
Throws:
SemanticException

setKeyAndValueDesc

public static void setKeyAndValueDesc(mapredWork plan,
                                      Operator<? extends Serializable> topOp)
set key and value descriptor

Parameters:
plan - current plan
topOp - current top operator in the path

getMapRedWork

public static mapredWork getMapRedWork()
create a new plan and return

Returns:
the new plan

splitTasks

public static void splitTasks(Operator<? extends Serializable> op,
                              Task<? extends Serializable> parentTask,
                              Task<? extends Serializable> childTask,
                              GenMRProcContext opProcCtx,
                              boolean setReducer,
                              boolean local,
                              int posn)
                       throws SemanticException
Throws:
SemanticException

mergeMapJoinUnion

public static void mergeMapJoinUnion(UnionOperator union,
                                     GenMRProcContext ctx,
                                     int pos)
                              throws SemanticException
Throws:
SemanticException


Copyright © 2009 The Apache Software Foundation