Accumulator

A simpler value of Accumulable where the result type being accumulated is the same as the types of elements being merged, i.e. variables that are only "added" to through an associative operation and can therefore be efficiently supported in parallel. They can be used to implement counters (as in MapReduce) or sums. Spark natively supports accumulators of numeric value types, and programmers can add support for new types.

An accumulator is created from an initial value v by calling SparkContext#accumulator. Tasks running on the cluster can then add to it using the Accumulable#+= operator. However, they cannot read its value. Only the driver program can read the accumulator's value, using its value method.

The interpreter session below shows an accumulator being used to add up the elements of an array:

scala> val accum = sc.accumulator(0)
accum: spark.Accumulator[Int] = 0

scala> sc.parallelize(Array(1, 2, 3, 4)).foreach(x => accum += x)
...
10/09/29 18:41:08 INFO SparkContext: Tasks finished in 0.317106 s

scala> accum.value
res2: Int = 10

T: result type

Linear Supertypes

Accumulable[T, T], Serializable, AnyRef, Any

Instance Constructors

new Accumulator(initialValue: T, param: AccumulatorParam[T])
new Accumulator(initialValue: T, param: AccumulatorParam[T], name: Option[String])

Value Members

final def !=(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def !=(arg0: Any): Boolean

Definition Classes
Any
final def ##(): Int

Definition Classes
AnyRef → Any
def ++=(term: T): Unit

Merge two accumulable objects together
Merge two accumulable objects together
Normally, a user will not want to use this version, but will instead call +=.
term
the other R that will get merged with this

Definition Classes
Accumulable
def +=(term: T): Unit

Add more data to this accumulator / accumulable
Add more data to this accumulator / accumulable
term
the data to add

Definition Classes
Accumulable
final def ==(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def ==(arg0: Any): Boolean

Definition Classes
Any
def add(term: T): Unit

Add more data to this accumulator / accumulable
Add more data to this accumulator / accumulable
term
the data to add

Definition Classes
Accumulable
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( ... )
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[java.lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
val id: Long

Definition Classes
Accumulable
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def localValue: T

Get the current value of this accumulator from within a task.
Get the current value of this accumulator from within a task.
This is NOT the global value of the accumulator. To get the global value after a completed operation on the dataset, call value.
The typical use of this method is to directly mutate the local value, eg., to add an element to a Set.

Definition Classes
Accumulable
def merge(term: T): Unit

Merge two accumulable objects together
Merge two accumulable objects together
Normally, a user will not want to use this version, but will instead call add.
term
the other R that will get merged with this

Definition Classes
Accumulable
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def setValue(newValue: T): Unit

Set the accumulator's value; only allowed on master
Set the accumulator's value; only allowed on master

Definition Classes
Accumulable
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
Accumulable → AnyRef → Any
def value: T

Access the accumulator's current value; only allowed on master.
Access the accumulator's current value; only allowed on master.

Definition Classes
Accumulable
def value_=(newValue: T): Unit

Set the accumulator's value; only allowed on master.
Set the accumulator's value; only allowed on master.

Definition Classes
Accumulable
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
val zero: T

Definition Classes
Accumulable

class Accumulator[T] extends Accumulable[T, T]

Instance Constructors

new Accumulator(initialValue: T, param: AccumulatorParam[T])

new Accumulator(initialValue: T, param: AccumulatorParam[T], name: Option[String])

Value Members

final def !=(arg0: AnyRef): Boolean

final def !=(arg0: Any): Boolean

final def ##(): Int

def ++=(term: T): Unit

def +=(term: T): Unit

final def ==(arg0: AnyRef): Boolean

final def ==(arg0: Any): Boolean

def add(term: T): Unit

final def asInstanceOf[T0]: T0

def clone(): AnyRef

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): Class[_]

def hashCode(): Int

val id: Long

final def isInstanceOf[T0]: Boolean

def localValue: T

def merge(term: T): Unit

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def setValue(newValue: T): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def value: T

def value_=(newValue: T): Unit

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

val zero: T

Inherited from Accumulable[T, T]

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped