org.apache.spark

Class MapOutputTracker

Object

org.apache.spark.MapOutputTracker

All Implemented Interfaces:

Logging

Direct Known Subclasses:

MapOutputTrackerMaster, MapOutputTrackerWorker

public abstract class MapOutputTracker
extends Object
implements Logging

Class that keeps track of the location of the map output of a stage. This is abstract because different versions of MapOutputTracker (driver and executor) use different HashMap to store its metadata.

Constructor Summary

Constructors

Constructor and Description
MapOutputTracker(SparkConf conf)

Method Summary

Methods

Modifier and Type	Method and Description
static MapStatus[]	deserializeMapStatuses(byte[] bytes)
long	getEpoch() Called to get current epoch number.
scala.Tuple2<BlockManagerId,Object>[]	getServerStatuses(int shuffleId, int reduceId) Called from executors to get the server URIs and output sizes of the map outputs of a given shuffle.
static byte[]	serializeMapStatuses(MapStatus[] statuses)
void	stop() Stop the tracker.
akka.actor.ActorRef	trackerActor() Set to the MapOutputTrackerActor living on the driver.
void	unregisterShuffle(int shuffleId) Unregister shuffle data.
void	updateEpoch(long newEpoch) Called from executors to update the epoch number, potentially clearing old outputs because of a fetch failure.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

MapOutputTracker

public MapOutputTracker(SparkConf conf)

Method Detail

serializeMapStatuses

public static byte[] serializeMapStatuses(MapStatus[] statuses)

deserializeMapStatuses

public static MapStatus[] deserializeMapStatuses(byte[] bytes)

trackerActor

public akka.actor.ActorRef trackerActor()

Set to the MapOutputTrackerActor living on the driver.

getServerStatuses

public scala.Tuple2<BlockManagerId,Object>[] getServerStatuses(int shuffleId,
                                                      int reduceId)

Called from executors to get the server URIs and output sizes of the map outputs of a given shuffle.

getEpoch

public long getEpoch()

Called to get current epoch number.

updateEpoch

public void updateEpoch(long newEpoch)

Called from executors to update the epoch number, potentially clearing old outputs because of a fetch failure. Each executor task calls this with the latest epoch number on the driver at the time it was created.

unregisterShuffle

public void unregisterShuffle(int shuffleId)

Unregister shuffle data.

stop

public void stop()

Stop the tracker.