org.apache.spark

Class MapOutputTrackerMaster

Object

org.apache.spark.MapOutputTracker

org.apache.spark.MapOutputTrackerMaster

All Implemented Interfaces:

Logging

public class MapOutputTrackerMaster
extends MapOutputTracker

MapOutputTracker for the driver. This uses TimeStampedHashMap to keep track of map output information, which allows old output information based on a TTL.

Constructor Summary

Constructors

Constructor and Description
MapOutputTrackerMaster(SparkConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	containsShuffle(int shuffleId) Check if the given shuffle is being tracked
byte[]	getSerializedMapOutputStatuses(int shuffleId)
void	incrementEpoch()
void	registerMapOutput(int shuffleId, int mapId, MapStatus status)
void	registerMapOutputs(int shuffleId, MapStatus[] statuses, boolean changeEpoch) Register multiple map output information for the given shuffle
void	registerShuffle(int shuffleId, int numMaps)
void	stop() Stop the tracker.
void	unregisterMapOutput(int shuffleId, int mapId, BlockManagerId bmAddress) Unregister map output information of the given shuffle, mapper and block manager
void	unregisterShuffle(int shuffleId) Unregister shuffle data

Methods inherited from class org.apache.spark.MapOutputTracker

deserializeMapStatuses, getEpoch, getServerStatuses, serializeMapStatuses, trackerActor, updateEpoch

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

MapOutputTrackerMaster

public MapOutputTrackerMaster(SparkConf conf)

Method Detail

registerShuffle

public void registerShuffle(int shuffleId,
                   int numMaps)

registerMapOutput

public void registerMapOutput(int shuffleId,
                     int mapId,
                     MapStatus status)

registerMapOutputs

public void registerMapOutputs(int shuffleId,
                      MapStatus[] statuses,
                      boolean changeEpoch)

Register multiple map output information for the given shuffle

unregisterMapOutput

public void unregisterMapOutput(int shuffleId,
                       int mapId,
                       BlockManagerId bmAddress)

Unregister map output information of the given shuffle, mapper and block manager

unregisterShuffle

public void unregisterShuffle(int shuffleId)

Unregister shuffle data

Overrides:

unregisterShuffle in class MapOutputTracker

containsShuffle

public boolean containsShuffle(int shuffleId)

Check if the given shuffle is being tracked

incrementEpoch

public void incrementEpoch()

getSerializedMapOutputStatuses

public byte[] getSerializedMapOutputStatuses(int shuffleId)

stop

public void stop()

Description copied from class: MapOutputTracker

Stop the tracker.

Overrides:

stop in class MapOutputTracker