AppStatusListener¶

AppStatusListener is a SparkListener.

Creating Instance¶

AppStatusListener takes the following to be created:

• ElementTrackingStore
• SparkConf
• live flag
• AppStatusSource (default: None)
• Last Update Time (default: None)

AppStatusListener is created when:

• AppStatusStore is requested to createLiveStore (with the live flag enabled)
• FsHistoryProvider is requested to rebuildAppStore (with the live flag disabled)

live Flag¶

AppStatusListener is given a live flag when created.

live flag indicates whether AppStatusListener is created for the following:

• true when created for a active (live) Spark application (for AppStatusStore)
• false when created for Spark History Server (for FsHistoryProvider)

onJobStart¶

onJobStart(
  event: SparkListenerJobStart): Unit

onJobStart...FIXME

onJobStart is part of the SparkListener abstraction.

onStageSubmitted¶

onStageSubmitted(
  event: SparkListenerStageSubmitted): Unit

onStageSubmitted is part of the SparkListener abstraction.

onStageSubmitted getOrCreateStage for the StageInfo (from the given SparkListenerStageSubmitted event).

onStageSubmitted...FIXME

In the end, onStageSubmitted liveUpdate with the stage and now timestamp.

Updating ElementTrackingStore for Active Spark Application¶

liveUpdate(
  entity: LiveEntity,
  now: Long): Unit

liveUpdate update the ElementTrackingStore when the live flag is enabled.

Updating ElementTrackingStore¶

update(
  entity: LiveEntity,
  now: Long,
  last: Boolean = false): Unit

update requests the given LiveEntity to write (with the ElementTrackingStore and checkTriggers flag being the given last flag).

getOrCreateStage¶

getOrCreateStage(
  info: StageInfo): LiveStage

getOrCreateStage...FIXME

getOrCreateStage is used when AppStatusListener is requested to onJobStart and onStageSubmitted.