ExecutorAllocationClient¶

ExecutorAllocationClient is an abstraction of schedulers that can communicate with a cluster manager to request or kill executors.

Implementations¶

• CoarseGrainedSchedulerBackend
• KubernetesClusterSchedulerBackend (Spark on Kubernetes)
• MesosCoarseGrainedSchedulerBackend
• StandaloneSchedulerBackend
• YarnSchedulerBackend

Active Executors¶

getExecutorIds(): Seq[String]

Used when:

• SparkContext is requested for active executors

Whether Executor is Active¶

isExecutorActive(
  id: String): Boolean

Returns whether a given executor is active (and can be used to execute tasks)

Killing Executors¶

killExecutors(
  executorIds: Seq[String],
  adjustTargetNumExecutors: Boolean,
  countFailures: Boolean,
  force: Boolean = false): Seq[String]

Requests a cluster manager to kill given executors and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when:

• ExecutorAllocationClient is requested to kill an executor
• ExecutorAllocationManager is requested to removeExecutors
• SparkContext is requested to kill executors and killAndReplaceExecutor
• BlacklistTracker is requested to kill an executor
• DriverEndpoint is requested to handle a KillExecutorsOnHost message

Killing Executors on Host¶

killExecutorsOnHost(
  host: String): Boolean

Used when:

• BlacklistTracker is requested to kill executors on a blacklisted node

Requesting Additional Executors¶

requestExecutors(
  numAdditionalExecutors: Int): Boolean

Requests additional executors from a cluster manager and returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when:

• SparkContext is requested for additional executors

Updating Total Executors¶

requestTotalExecutors(
  numExecutors: Int,
  localityAwareTasks: Int,
  hostToLocalTaskCount: Map[String, Int]): Boolean

Updates a cluster manager with the exact number of executors desired. Returns whether the request has been acknowledged by the cluster manager (true) or not (false).

Used when:

• SparkContext is requested to update the number of total executors

• ExecutorAllocationManager is requested to start, updateAndSyncNumExecutorsTarget, addExecutors, removeExecutors

Killing Executor¶

killExecutor(
  executorId: String): Boolean

killExecutor kill the given executor.

killExecutor is used when:

• ExecutorAllocationManager removes an executor.
• SparkContext is requested to kill executors.