PluginContainer¶

PluginContainer is an abstraction of plugin containers that can registerMetrics (for the driver and executors).

PluginContainer is created for the driver and executors using apply utility.

Contract¶

registerMetrics¶

registerMetrics(
  appId: String): Unit

Used when SparkContext is created

shutdown¶

shutdown(): Unit

Used when:

SparkContext is requested to stop
Executor is requested to stop

Implementations¶

Sealed Abstract Class

PluginContainer is a Scala sealed abstract class which means that all of the implementations are in the same compilation unit (a single file).

Creating PluginContainer¶

// the driver
apply(
  sc: SparkContext,
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]
// executors
apply(
  env: SparkEnv,
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]
// private helper
apply(
  ctx: Either[SparkContext, SparkEnv],
  resources: java.util.Map[String, ResourceInformation]): Option[PluginContainer]

apply creates a PluginContainer for the driver or executors (based on the type of the first input argument, i.e. SparkContext or SparkEnv, respectively).

apply first loads the SparkPlugins defined by spark.plugins configuration property.

Only when there was at least one plugin loaded, apply creates a DriverPluginContainer or ExecutorPluginContainer.

apply is used when:

SparkContext is created
Executor is created

Last update: 2020-10-08