@Private
public interface ShuffleExecutorComponents
Modifier and Type | Method and Description |
---|---|
ShuffleMapOutputWriter |
createMapOutputWriter(int shuffleId,
long mapTaskId,
int numPartitions)
Called once per map task to create a writer that will be responsible for persisting all the
partitioned bytes written by that map task.
|
default java.util.Optional<SingleSpillShuffleMapOutputWriter> |
createSingleFileMapOutputWriter(int shuffleId,
long mapId)
An optional extension for creating a map output writer that can optimize the transfer of a
single partition file, as the entire result of a map task, to the backing store.
|
void |
initializeExecutor(String appId,
String execId,
java.util.Map<String,String> extraConfigs)
Called once per executor to bootstrap this module with state that is specific to
that executor, specifically the application ID and executor ID.
|
void initializeExecutor(String appId, String execId, java.util.Map<String,String> extraConfigs)
appId
- The Spark application idexecId
- The unique identifier of the executor being initializedextraConfigs
- Extra configs that were returned by
ShuffleDriverComponents.initializeApplication()
ShuffleMapOutputWriter createMapOutputWriter(int shuffleId, long mapTaskId, int numPartitions) throws java.io.IOException
shuffleId
- Unique identifier for the shuffle the map task is a part ofmapTaskId
- An ID of the map task. The ID is unique within this Spark application.numPartitions
- The number of partitions that will be written by the map task. Some of
these partitions may be empty.java.io.IOException
default java.util.Optional<SingleSpillShuffleMapOutputWriter> createSingleFileMapOutputWriter(int shuffleId, long mapId) throws java.io.IOException
Most implementations should return the default Optional.empty()
to indicate that
they do not support this optimization. This primarily is for backwards-compatibility in
preserving an optimization in the local disk shuffle storage implementation.
shuffleId
- Unique identifier for the shuffle the map task is a part ofmapId
- An ID of the map task. The ID is unique within this Spark application.java.io.IOException