org.apache.spark

Class ShuffleDependency<K,V,C>

Object

org.apache.spark.Dependency<scala.Product2<K,V>>

org.apache.spark.ShuffleDependency<K,V,C>

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.internal.Logging

public class ShuffleDependency<K,V,C>
extends Dependency<scala.Product2<K,V>>
implements org.apache.spark.internal.Logging

:: DeveloperApi :: Represents a dependency on the output of a shuffle stage. Note that in the case of shuffle, the RDD is transient since we don't need it on the executor side.

param: _rdd the parent RDD param: partitioner partitioner used to partition the shuffle output param: serializer Serializer to use. If not set explicitly then the default serializer, as specified by spark.serializer config option, will be used. param: keyOrdering key ordering for RDD's shuffles param: aggregator map/reduce-side aggregator for RDD's shuffle param: mapSideCombine whether to perform partial aggregation (also known as map-side combine) param: shuffleWriterProcessor the processor to control the write behavior in ShuffleMapTask

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description

ShuffleDependency(RDD<? extends scala.Product2<K,V>> _rdd, Partitioner partitioner, Serializer serializer, scala.Option<scala.math.Ordering<K>> keyOrdering, scala.Option<Aggregator<K,V,C>> aggregator, boolean mapSideCombine, org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor, scala.reflect.ClassTag<K> evidence$1, scala.reflect.ClassTag<V> evidence$2, scala.reflect.ClassTag<C> evidence$3)

Method Summary

Modifier and Type	Method and Description
scala.Option<Aggregator<K,V,C>>	aggregator()
scala.collection.Seq<BlockManagerId>	getMergerLocs()
scala.Option<scala.math.Ordering<K>>	keyOrdering()
boolean	mapSideCombine()
void	newShuffleMergeState()
Partitioner	partitioner()
RDD<scala.Product2<K,V>>	rdd()
Serializer	serializer()
void	setMergerLocs(scala.collection.Seq<BlockManagerId> mergerLocs)
org.apache.spark.shuffle.ShuffleHandle	shuffleHandle()
int	shuffleId()
boolean	shuffleMergeEnabled()
boolean	shuffleMergeFinalized() Returns true if push-based shuffle is disabled for this stage or empty RDD, or if the shuffle merge for this stage is finalized, i.e.
int	shuffleMergeId() shuffleMergeId is used to uniquely identify merging process of shuffle by an indeterminate stage attempt.
org.apache.spark.shuffle.ShuffleWriteProcessor	shuffleWriterProcessor()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize

Constructor Detail

ShuffleDependency

public ShuffleDependency(RDD<? extends scala.Product2<K,V>> _rdd,
                         Partitioner partitioner,
                         Serializer serializer,
                         scala.Option<scala.math.Ordering<K>> keyOrdering,
                         scala.Option<Aggregator<K,V,C>> aggregator,
                         boolean mapSideCombine,
                         org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor,
                         scala.reflect.ClassTag<K> evidence$1,
                         scala.reflect.ClassTag<V> evidence$2,
                         scala.reflect.ClassTag<C> evidence$3)

Method Detail

partitioner

public Partitioner partitioner()

serializer

public Serializer serializer()

keyOrdering

public scala.Option<scala.math.Ordering<K>> keyOrdering()

aggregator

public scala.Option<Aggregator<K,V,C>> aggregator()

mapSideCombine

public boolean mapSideCombine()

shuffleWriterProcessor

public org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor()

rdd

public RDD<scala.Product2<K,V>> rdd()

Specified by:

rdd in class Dependency<scala.Product2<K,V>>

shuffleId

public int shuffleId()

shuffleHandle

public org.apache.spark.shuffle.ShuffleHandle shuffleHandle()

shuffleMergeEnabled

public boolean shuffleMergeEnabled()

shuffleMergeId

public int shuffleMergeId()

shuffleMergeId is used to uniquely identify merging process of shuffle by an indeterminate stage attempt.

Returns:

(undocumented)

setMergerLocs

public void setMergerLocs(scala.collection.Seq<BlockManagerId> mergerLocs)

getMergerLocs

public scala.collection.Seq<BlockManagerId> getMergerLocs()

shuffleMergeFinalized

public boolean shuffleMergeFinalized()

Returns true if push-based shuffle is disabled for this stage or empty RDD, or if the shuffle merge for this stage is finalized, i.e. the shuffle merge results for all partitions are available.

Returns:

(undocumented)

newShuffleMergeState

public void newShuffleMergeState()