| Interface | Description |
|---|---|
| DAGSchedulerEvent |
Types of events that can be handled by the DAGScheduler.
|
| JobListener |
Interface used to listen for job completion or failure events after submitting a job to the
DAGScheduler.
|
| JobResult |
:: DeveloperApi ::
A result of a job in the DAGScheduler.
|
| MapStatus |
Result returned by a ShuffleMapTask to a scheduler.
|
| OutputCommitCoordinationMessage | |
| Schedulable |
An interface for schedulable entities.
|
| SchedulableBuilder |
An interface to build Schedulable tree
buildPools: build the tree nodes(pools)
addTaskSetManager: build the leaf nodes(TaskSetManagers)
|
| SchedulerBackend |
A backend interface for scheduling systems that allows plugging in different ones under
TaskSchedulerImpl.
|
| SchedulingAlgorithm |
An interface for sort algorithm
FIFO: FIFO algorithm between TaskSetManagers
FS: FS algorithm between Pools, and FIFO or FS within Pools
|
| SparkListener |
:: DeveloperApi ::
Interface for listening to events from the Spark scheduler.
|
| SparkListenerBus |
A
SparkListenerEvent bus that relays SparkListenerEvents to its listeners |
| SparkListenerEvent | |
| TaskLocation |
A location where a task should run.
|
| TaskResult<T> | |
| TaskScheduler |
Low-level task scheduler interface, currently implemented exclusively by TaskSchedulerImpl.
|
| Class | Description |
|---|---|
| AccumulableInfo |
:: DeveloperApi ::
Information about an
Accumulable modified during a task or stage. |
| ActiveJob |
Tracks information about an active job in the DAGScheduler.
|
| AllJobsCancelled | |
| ApplicationEventListener |
A simple listener for application events.
|
| AskPermissionToCommitOutput | |
| BeginEvent | |
| CompletionEvent | |
| CompressedMapStatus |
A
MapStatus implementation that tracks the size of each block. |
| DAGScheduler |
The high-level scheduling layer that implements stage-oriented scheduling.
|
| DAGSchedulerEventProcessLoop | |
| DAGSchedulerSource | |
| DirectTaskResult<T> |
A TaskResult that contains the task's return value and accumulator updates.
|
| EventLoggingListener |
A SparkListener that logs events to persistent storage.
|
| ExecutorAdded | |
| ExecutorCacheTaskLocation |
A location that includes both a host and an executor id on that host.
|
| ExecutorExited | |
| ExecutorLossReason |
Represents an explanation for a executor or whole slave failing or exiting.
|
| ExecutorLost | |
| FairSchedulableBuilder | |
| FairSchedulingAlgorithm | |
| FIFOSchedulableBuilder | |
| FIFOSchedulingAlgorithm | |
| GettingResultEvent | |
| HDFSCacheTaskLocation |
A location on a host that is cached by HDFS.
|
| HighlyCompressedMapStatus |
A
MapStatus implementation that only stores the average size of non-empty blocks,
plus a bitmap for tracking which blocks are empty. |
| HostTaskLocation |
A location on a host.
|
| IndirectTaskResult<T> |
A reference to a DirectTaskResult that has been stored in the worker's BlockManager.
|
| InputFormatInfo |
:: DeveloperApi ::
Parses and holds information about inputFormat (and files) specified as a parameter.
|
| JobCancelled | |
| JobFailed | |
| JobGroupCancelled | |
| JobLogger |
:: DeveloperApi ::
A logger class to record runtime information for jobs in Spark.
|
| JobSubmitted | |
| JobSucceeded | |
| JobWaiter<T> |
An object that waits for a DAGScheduler job to complete.
|
| LiveListenerBus |
Asynchronously passes SparkListenerEvents to registered SparkListeners.
|
| OutputCommitCoordinator |
Authority that decides whether tasks can commit output to HDFS.
|
| OutputCommitCoordinator.OutputCommitCoordinatorActor | |
| Pool |
An Schedulable entity that represent collection of Pools or TaskSetManagers
|
| ReplayListenerBus |
A SparkListenerBus that can be used to replay events from serialized event data.
|
| ResubmitFailedStages | |
| ResultTask<T,U> |
A task that sends back the output to the driver application.
|
| RuntimePercentage | |
| SchedulingMode |
"FAIR" and "FIFO" determines which policy is used
to order tasks amongst a Schedulable's sub-queues
"NONE" is used when the a Schedulable has no sub-queues.
|
| ShuffleMapTask |
A ShuffleMapTask divides the elements of an RDD into multiple buckets (based on a partitioner
specified in the ShuffleDependency).
|
| SlaveLost | |
| SparkListenerApplicationEnd | |
| SparkListenerApplicationStart | |
| SparkListenerBlockManagerAdded | |
| SparkListenerBlockManagerRemoved | |
| SparkListenerEnvironmentUpdate | |
| SparkListenerExecutorAdded | |
| SparkListenerExecutorMetricsUpdate |
Periodic updates from executors.
|
| SparkListenerExecutorRemoved | |
| SparkListenerJobEnd | |
| SparkListenerJobStart | |
| SparkListenerLogStart |
An internal class that describes the metadata of an event log.
|
| SparkListenerStageCompleted | |
| SparkListenerStageSubmitted | |
| SparkListenerTaskEnd | |
| SparkListenerTaskGettingResult | |
| SparkListenerTaskStart | |
| SparkListenerUnpersistRDD | |
| SplitInfo | |
| Stage |
A stage is a set of independent tasks all computing the same function that need to run as part
of a Spark job, where all the tasks have the same shuffle dependencies.
|
| StageCancelled | |
| StageInfo |
:: DeveloperApi ::
Stores information about a stage to pass from the scheduler to SparkListeners.
|
| StatsReportListener |
:: DeveloperApi ::
Simple SparkListener that logs a few summary statistics when each stage completes
|
| StopCoordinator | |
| Task<T> |
A unit of execution.
|
| TaskDescription |
Description of a task that gets passed onto executors to be executed, usually created by
TaskSetManager.resourceOffer. |
| TaskInfo |
:: DeveloperApi ::
Information about a running task attempt inside a TaskSet.
|
| TaskLocality | |
| TaskResultGetter |
Runs a thread pool that deserializes and remotely fetches (if necessary) task results.
|
| TaskSchedulerImpl |
Schedules tasks for multiple types of clusters by acting through a SchedulerBackend.
|
| TaskSet |
A set of tasks submitted together to the low-level TaskScheduler, usually representing
missing partitions of a particular stage.
|
| TaskSetFailed | |
| TaskSetManager |
Schedules the tasks within a single TaskSet in the TaskSchedulerImpl.
|
| WorkerOffer |
Represents free resources available on an executor.
|