org.apache.pig.backend.hadoop.executionengine
Class HExecutionEngine

java.lang.Object
  extended by org.apache.pig.backend.hadoop.executionengine.HExecutionEngine
All Implemented Interfaces:
ExecutionEngine

public class HExecutionEngine
extends Object
implements ExecutionEngine


Field Summary
protected  DataStorage ds
           
static String JOB_TRACKER_LOCATION
           
protected  org.apache.hadoop.mapred.JobClient jobClient
           
protected  Map<OperatorKey,OperatorKey> logicalToPhysicalKeys
           
protected  Map<OperatorKey,MapRedResult> materializedResults
           
protected  Map<OperatorKey,ExecPhysicalOperator> physicalOpTable
           
protected  PigContext pigContext
           
 
Constructor Summary
HExecutionEngine(PigContext pigContext)
           
 
Method Summary
 Collection<String> activeScopes()
          List scopes that are active in the back-end
 void close()
          Clean-up and releasing of resources.
 PhysicalPlan compile(LogicalPlan plan, Properties properties)
          Compiles a logical plan into a physical plan, given a set of configuration properties that apply at the plan-level.
 List<ExecJob> execute(PhysicalPlan plan, String jobName)
          Execute the physical plan in blocking mode.
 void explain(PhysicalPlan plan, PrintStream stream, String format, boolean verbose)
          Explain executor specific information.
 Properties getConfiguration()
          Provides configuration information about the execution engine itself.
 DataStorage getDataStorage()
           
 org.apache.hadoop.mapred.JobClient getJobClient()
           
 Map<OperatorKey,MapRedResult> getMaterializedResults()
           
 Map<OperatorKey,ExecPhysicalOperator> getPhysicalOpTable()
           
 Map<String,Object> getStatistics()
          Provides statistics on the Execution Engine: number of nodes, node failure rates, average load, average job wait time...
 void init()
          Place holder for possible initialization activities.
 void init(Properties properties)
           
 void reclaimScope(String scope)
          A mechanism to communicate to the back-end that a set of logical plans go out of scope
 Collection<ExecJob> runningJobs(Properties properties)
          Return currently running jobs (can be useful for admin purposes)
 List<ExecJob> submit(PhysicalPlan plan, String jobName)
          Execute the physical plan in non-blocking mode
 void updateConfiguration(Properties newConfiguration)
          Provides a way to dynamically change configuration parameters at the Execution Engine level.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

JOB_TRACKER_LOCATION

public static final String JOB_TRACKER_LOCATION
See Also:
Constant Field Values

pigContext

protected PigContext pigContext

ds

protected DataStorage ds

jobClient

protected org.apache.hadoop.mapred.JobClient jobClient

logicalToPhysicalKeys

protected Map<OperatorKey,OperatorKey> logicalToPhysicalKeys

physicalOpTable

protected Map<OperatorKey,ExecPhysicalOperator> physicalOpTable

materializedResults

protected Map<OperatorKey,MapRedResult> materializedResults
Constructor Detail

HExecutionEngine

public HExecutionEngine(PigContext pigContext)
Method Detail

getJobClient

public org.apache.hadoop.mapred.JobClient getJobClient()

getMaterializedResults

public Map<OperatorKey,MapRedResult> getMaterializedResults()

getPhysicalOpTable

public Map<OperatorKey,ExecPhysicalOperator> getPhysicalOpTable()

getDataStorage

public DataStorage getDataStorage()
Specified by:
getDataStorage in interface ExecutionEngine

init

public void init()
          throws ExecException
Description copied from interface: ExecutionEngine
Place holder for possible initialization activities.

Specified by:
init in interface ExecutionEngine
Throws:
ExecException

init

public void init(Properties properties)
          throws ExecException
Throws:
ExecException

close

public void close()
           throws ExecException
Description copied from interface: ExecutionEngine
Clean-up and releasing of resources.

Specified by:
close in interface ExecutionEngine
Throws:
ExecException

getConfiguration

public Properties getConfiguration()
                            throws ExecException
Description copied from interface: ExecutionEngine
Provides configuration information about the execution engine itself.

Specified by:
getConfiguration in interface ExecutionEngine
Returns:
- information about the configuration used to connect to execution engine
Throws:
ExecException

updateConfiguration

public void updateConfiguration(Properties newConfiguration)
                         throws ExecException
Description copied from interface: ExecutionEngine
Provides a way to dynamically change configuration parameters at the Execution Engine level.

Specified by:
updateConfiguration in interface ExecutionEngine
Parameters:
newConfiguration - - the new configuration settings
Throws:
ExecException

getStatistics

public Map<String,Object> getStatistics()
                                 throws ExecException
Description copied from interface: ExecutionEngine
Provides statistics on the Execution Engine: number of nodes, node failure rates, average load, average job wait time...

Specified by:
getStatistics in interface ExecutionEngine
Returns:
ExecutionEngineProperties
Throws:
ExecException

compile

public PhysicalPlan compile(LogicalPlan plan,
                            Properties properties)
                     throws ExecException
Description copied from interface: ExecutionEngine
Compiles a logical plan into a physical plan, given a set of configuration properties that apply at the plan-level. For instance desired degree of parallelism for this plan, which could be different from the "default" one set at the execution engine level.

Specified by:
compile in interface ExecutionEngine
Parameters:
plan - logical plan to compile
Returns:
physical plan
Throws:
ExecException

execute

public List<ExecJob> execute(PhysicalPlan plan,
                             String jobName)
                      throws ExecException
Description copied from interface: ExecutionEngine
Execute the physical plan in blocking mode.

Specified by:
execute in interface ExecutionEngine
Parameters:
plan - PhysicalPlan to execute.
jobName - Name of this plan, will be used to identify the plan
Throws:
ExecException

submit

public List<ExecJob> submit(PhysicalPlan plan,
                            String jobName)
                     throws ExecException
Description copied from interface: ExecutionEngine
Execute the physical plan in non-blocking mode

Specified by:
submit in interface ExecutionEngine
Parameters:
plan - PhysicalPlan to submit.
jobName - Name of this plan, will be used to identify the plan
Throws:
ExecException

explain

public void explain(PhysicalPlan plan,
                    PrintStream stream,
                    String format,
                    boolean verbose)
Description copied from interface: ExecutionEngine
Explain executor specific information.

Specified by:
explain in interface ExecutionEngine
Parameters:
plan - PhysicalPlan to explain
stream - Stream to print output to
format - Format to print in
verbose - Amount of information to print

runningJobs

public Collection<ExecJob> runningJobs(Properties properties)
                                throws ExecException
Description copied from interface: ExecutionEngine
Return currently running jobs (can be useful for admin purposes)

Specified by:
runningJobs in interface ExecutionEngine
Returns:
All jobs that are currently active in hte execution engine.
Throws:
ExecException

activeScopes

public Collection<String> activeScopes()
                                throws ExecException
Description copied from interface: ExecutionEngine
List scopes that are active in the back-end

Specified by:
activeScopes in interface ExecutionEngine
Returns:
All scopes that are currently active in the execution engine.
Throws:
ExecException

reclaimScope

public void reclaimScope(String scope)
                  throws ExecException
Description copied from interface: ExecutionEngine
A mechanism to communicate to the back-end that a set of logical plans go out of scope

Specified by:
reclaimScope in interface ExecutionEngine
Throws:
ExecException


Copyright © ${year} The Apache Software Foundation