AccumulatorV2¶

AccumulatorV2[IN, OUT] is an abstraction of accumulators

AccumulatorV2 is a Java Serializable.

Contract¶

Adding Value¶

add(
  v: IN): Unit

Accumulates (adds) the given v value to this accumulator

Copying Accumulator¶

copy(): AccumulatorV2[IN, OUT]

Is Zero Value¶

isZero: Boolean

Merging Updates¶

merge(
  other: AccumulatorV2[IN, OUT]): Unit

Resetting Accumulator¶

reset(): Unit

Value¶

value: OUT

The current value of this accumulator

Used when:

• TaskRunner is requested to collectAccumulatorsAndResetStatusOnFailure
• AccumulatorSource is requested to register
• DAGScheduler is requested to update accumulators
• TaskSchedulerImpl is requested to executorHeartbeatReceived
• TaskSetManager is requested to handleSuccessfulTask
• JsonProtocol is requested to taskEndReasonFromJson
• others

Implementations¶

• AggregatingAccumulator (Spark SQL)
• CollectionAccumulator
• DoubleAccumulator
• EventTimeStatsAccum (Spark Structured Streaming)
• LongAccumulator
• SetAccumulator (Spark SQL)
• SQLMetric (Spark SQL)

Converting this Accumulator to AccumulableInfo¶

toInfo(
  update: Option[Any],
  value: Option[Any]): AccumulableInfo

toInfo determines whether the accumulator is internal based on the name (and whether it uses the internal.metrics prefix) and uses it to create an AccumulableInfo.

toInfo is used when:

• TaskRunner is requested to collectAccumulatorsAndResetStatusOnFailure
• DAGScheduler is requested to updateAccumulators
• TaskSchedulerImpl is requested to executorHeartbeatReceived
• JsonProtocol is requested to taskEndReasonFromJson
• SQLAppStatusListener (Spark SQL) is requested to handle a SparkListenerTaskEnd event (onTaskEnd)

Registering Accumulator¶

register(
  sc: SparkContext,
  name: Option[String] = None,
  countFailedValues: Boolean = false): Unit

register...FIXME

register is used when:

• SparkContext is requested to register an accumulator
• TaskMetrics is requested to register task accumulators
• CollectMetricsExec (Spark SQL) is requested for an AggregatingAccumulator
• SQLMetrics (Spark SQL) is used to create a performance metric

writeReplace¶

writeReplace(): Any

writeReplace...FIXME