org.apache.spark

Class FetchFailed

Object

org.apache.spark.FetchFailed

All Implemented Interfaces:

java.io.Serializable, TaskEndReason, TaskFailedReason, scala.Equals, scala.Product

public class FetchFailed
extends Object
implements TaskFailedReason, scala.Product, scala.Serializable

:: DeveloperApi :: Task failed to fetch shuffle data from a remote node. Probably means we have lost the remote executors the task is trying to fetch from, and thus need to rerun the previous stage.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
FetchFailed(BlockManagerId bmAddress, int shuffleId, int mapId, int reduceId, String message)

Method Summary

Modifier and Type	Method and Description
abstract static R	apply(T1 v1, T2 v2, T3 v3, T4 v4, T5 v5)
BlockManagerId	bmAddress()
boolean	countTowardsTaskFailures() Fetch failures lead to a different failure handling path: (1) we don't abort the stage after 4 task failures, instead we immediately go back to the stage which generated the map output, and regenerate the missing data.
int	mapId()
String	message()
int	reduceId()
int	shuffleId()
String	toErrorString() Error message displayed in the web UI.
static String	toString()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface scala.Product

productArity, productElement, productIterator, productPrefix

Methods inherited from interface scala.Equals

canEqual, equals

Constructor Detail

FetchFailed

public FetchFailed(BlockManagerId bmAddress,
                   int shuffleId,
                   int mapId,
                   int reduceId,
                   String message)

Method Detail

apply

public abstract static R apply(T1 v1,
                               T2 v2,
                               T3 v3,
                               T4 v4,
                               T5 v5)

toString

public static String toString()

bmAddress

public BlockManagerId bmAddress()

shuffleId

public int shuffleId()

mapId

public int mapId()

reduceId

public int reduceId()

message

public String message()

toErrorString

public String toErrorString()

Description copied from interface: TaskFailedReason

Error message displayed in the web UI.

Specified by:

toErrorString in interface TaskFailedReason

countTowardsTaskFailures

public boolean countTowardsTaskFailures()

Fetch failures lead to a different failure handling path: (1) we don't abort the stage after 4 task failures, instead we immediately go back to the stage which generated the map output, and regenerate the missing data. (2) we don't count fetch failures for blacklisting, since presumably its not the fault of the executor where the task ran, but the executor which stored the data. This is especially important because we might rack up a bunch of fetch-failures in rapid succession, on all nodes of the cluster, due to one bad node.

Specified by:

countTowardsTaskFailures in interface TaskFailedReason

Returns:

(undocumented)