The Internals of Apache Spark

EnvironmentListener

Initializing search

spark-internals

Home
Internals
Spark Standalone
Monitoring
Tools
RDD
Metrics
Demos
Exercises
Web UI

The Internals of Apache Spark

spark-internals

Home
Internals
Internals
- Overview
- SparkEnv
- SparkConf
- SparkContext
  SparkContext
- SparkFiles
- Configuration Properties
- Spark Properties
- External Shuffle Service
  External Shuffle Service
- MapOutputTracker
  MapOutputTracker
  - MapOutputTracker
  - MapOutputTrackerMaster
    MapOutputTrackerMaster
    
    MapOutputTrackerMaster
    
    ShuffleStatus
    
    MapOutputTrackerMasterEndpoint
  - MapOutputTrackerWorker
- Shuffle System
  Shuffle System
  - ShuffleManager
  - SortShuffleManager
  - Shuffle Handlers
    Shuffle Handlers
    
    ShuffleHandle
    
    BaseShuffleHandle
    
    BypassMergeSortShuffleHandle
    
    SerializedShuffleHandle
  - Shuffle Readers
    Shuffle Readers
    
    ShuffleReader
    
    BlockStoreShuffleReader
  - Shuffle Writers
    Shuffle Writers
    
    ShuffleWriter
    
    BypassMergeSortShuffleWriter
    
    SortShuffleWriter
    
    UnsafeShuffleWriter
  - ShuffleExternalSorter
  - ShuffleInMemorySorter
  - ShuffleBlockResolver
  - IndexShuffleBlockResolver
  - FetchFailedException
  - Spillable Collections
    Spillable Collections
    
    Spillable
    
    ExternalAppendOnlyMap
    
    ExternalSorter
  - ShuffleWriteProcessor
  - ShuffleWriteMetricsReporter
  - ShuffleDataIOs
    ShuffleDataIOs
    
    ShuffleDataIO
    
    LocalDiskShuffleDataIO
    
    ShuffleDriverComponents
    
    ShuffleExecutorComponents
    
    ShuffleMapOutputWriter
    
    SingleSpillShuffleMapOutputWriter
    
    LocalDiskShuffleExecutorComponents
    
    ShuffleDataIOUtils
- Dynamic Resource Allocation
  Dynamic Resource Allocation
- Network
  Network
- Core
  Core
- Scheduler
  Scheduler
- RPC
  RPC
- Memory
  Memory
- Storage
  Storage
- Serialization
  Serialization
- Executor
  Executor
- Broadcast Variables
  Broadcast Variables
  - Broadcast
- Accumulators
  Accumulators
- Barrier Execution Mode
  Barrier Execution Mode
  - barrier-execution-mode
  - RDDBarrier
- Speculative Execution of Tasks
- Deployment Architecture
  Deployment Architecture
  - Architecture
  - Driver
  - Master
  - Workers
- Internal IO
  Internal IO
- Stage-Level Scheduling
  Stage-Level Scheduling
  - ResourceProfile
  - ResourceProfileBuilder
- Misc
  Misc
- Spark Tips and Tricks
  Spark Tips and Tricks
- Spark Local
  Spark Local
Spark Standalone
Spark Standalone
Monitoring
Monitoring
- Logging
- ListenerBus
- SparkListenerBus
- SparkListenerEvent
- SparkListener
- SparkListenerInterface
- SpillListener
- StatsReportListener
- AsyncEventQueue
- spark-debugging
- ConsoleProgressBar
- Spark History Server
  Spark History Server
- Status REST API
  Status REST API
- Plugin Framework
  Plugin Framework
Tools
Tools
- spark-submit
  spark-submit
- spark-shell
- spark-class
- SparkLauncher
- Internals
  Internals
RDD
RDD
- RDD
- Custom RDDs
  Custom RDDs
- Operators
  Operators
- Partitioners
  Partitioners
- RDD Lineage
- Caching and Persistence
- Partitions and Partitioning
- Partition
- RDD Checkpointing
- RDDCheckpointData
- LocalRDDCheckpointData
- ReliableRDDCheckpointData
- Shuffling
- Dependencies
- NarrowDependency
- ShuffleDependency
- Aggregator
Metrics
Metrics
- Configuration Properties
- MetricsSystem
- MetricsConfig
- Source
- Sink
- Sources
  Sources
  - JvmSource
- Sinks
  Sinks
  - MetricsServlet
Demos
Demos
- DiskBlockManager and Block Data
Exercises
Exercises
Web UI
Web UI
- Configuration Properties
- WebUI
- WebUIPage
- WebUITab
- SparkUI
- SparkUITab
- Jobs Tab
  Jobs Tab
  - Jobs
  - JobsTab
  - AllJobsPage
  - JobPage
- Stages Tab
  Stages Tab
  - Stages
  - StagesTab
  - AllStagesPage
  - StagePage
  - PoolPage
- Storage Tab
  Storage Tab
- Environment Tab
  Environment Tab
- Executors
  Executors
- BlockStatusListener
- EnvironmentListener
- ExecutorsListener
- JobProgressListener
- StorageStatusListener
- StorageListener
- RDDOperationGraphListener
- RDDStorageInfo
- LiveEntity
- LiveRDD
- UIUtils
- JettyUtils

EnvironmentListener

== [[EnvironmentListener]] EnvironmentListener Spark Listener

CAUTION: FIXME

Last update: 2020-10-06

Previous BlockStatusListener

Next ExecutorsListener

Copyright © 2021 Jacek Laskowski

Made with Material for MkDocs Insiders