Skip to content
The Internals of Apache Spark
CompressedMapStatus
Initializing search
spark-internals
Home
Internals
Spark Standalone
Monitoring
Tools
RDD
Metrics
Demos
Exercises
Web UI
The Internals of Apache Spark
spark-internals
Home
Internals
Internals
Overview
SparkEnv
SparkConf
SparkContext
SparkContext
SparkContext
Inside Creating SparkContext
SparkStatusTracker
Local Properties
SparkFiles
Configuration Properties
Spark Properties
External Shuffle Service
External Shuffle Service
Configuration Properties
ExternalShuffleService
ExternalBlockHandler
ExternalShuffleBlockResolver
ExecutorShuffleInfo
MapOutputTracker
MapOutputTracker
MapOutputTracker
MapOutputTrackerMaster
MapOutputTrackerMaster
MapOutputTrackerMaster
ShuffleStatus
MapOutputTrackerMasterEndpoint
MapOutputTrackerWorker
Shuffle System
Shuffle System
ShuffleManager
SortShuffleManager
Shuffle Handlers
Shuffle Handlers
ShuffleHandle
BaseShuffleHandle
BypassMergeSortShuffleHandle
SerializedShuffleHandle
Shuffle Readers
Shuffle Readers
ShuffleReader
BlockStoreShuffleReader
Shuffle Writers
Shuffle Writers
ShuffleWriter
BypassMergeSortShuffleWriter
SortShuffleWriter
UnsafeShuffleWriter
ShuffleExternalSorter
ShuffleInMemorySorter
ShuffleBlockResolver
IndexShuffleBlockResolver
FetchFailedException
Spillable Collections
Spillable Collections
Spillable
ExternalAppendOnlyMap
ExternalSorter
ShuffleWriteProcessor
ShuffleWriteMetricsReporter
ShuffleDataIOs
ShuffleDataIOs
ShuffleDataIO
LocalDiskShuffleDataIO
ShuffleDriverComponents
ShuffleExecutorComponents
ShuffleMapOutputWriter
SingleSpillShuffleMapOutputWriter
LocalDiskShuffleExecutorComponents
ShuffleDataIOUtils
Dynamic Resource Allocation
Dynamic Resource Allocation
Configuration Properties
ExecutorAllocationManager
ExecutorMonitor
Tracker
ExecutorAllocationClient
ExecutorAllocationManagerSource
ExecutorAllocationListener
Network
Network
TransportContext
MessageHandler
TransportRequestHandler
TransportConf
SparkTransportConf
TransportClientFactory
TransportServer
RpcHandler
StreamManager
OneForOneStreamManager
ManagedBuffer
RpcResponseCallback
Core
Core
BroadcastManager
BroadcastFactory
TorrentBroadcastFactory
TorrentBroadcast
ContextCleaner
CleanerListener
BlockFetchingListener
RetryingBlockFetcher
BlockFetchStarter
AppStatusListener
AppStatusStore
KVStore
ElementTrackingStore
InMemoryStore
LevelDB
Scheduler
Scheduler
DAGScheduler
Stage
ResultStage
ShuffleMapStage
BlacklistTracker
StageInfo
JobListener
JobWaiter
TaskScheduler
TaskSchedulerImpl
SchedulerBackend
CoarseGrainedSchedulerBackend
DriverEndpoint
SchedulerBackendUtils
Task
ShuffleMapTask
ResultTask
TaskSet
TaskSetManager
ActiveJob
SchedulableBuilder
FIFOSchedulableBuilder
FairSchedulableBuilder
Schedulable
Pool
SchedulingMode
TaskInfo
MapStatuses
MapStatuses
MapStatus
CompressedMapStatus
HighlyCompressedMapStatus
TaskDescription
TaskResultGetter
TaskContext
BarrierTaskContext
TaskContextImpl
TaskResult
TaskSetBlacklist
OutputCommitCoordinator
TaskLocation
ExternalClusterManager
LiveListenerBus
DAGSchedulerEvent
DAGSchedulerEventProcessLoop
DAGSchedulerSource
ExecutorData
RPC
RPC
RpcEnv
NettyRpcEnv
NettyStreamManager
RpcEnvConfig
RpcEndpoint
RpcEndpointRef
RpcAddress
RpcEndpointAddress
RpcEnvFactory
NettyRpcEnvFactory
RpcEnvFileServer
spark-rpc-netty
RpcUtils
Memory
Memory
MemoryManager
StaticMemoryManager
UnifiedMemoryManager
MemoryPool
ExecutionMemoryPool
StorageMemoryPool
TaskMemoryManager
MemoryConsumer
BytesToBytesMap
UnsafeSorterSpillReader
UnsafeExternalSorter
UnsafeInMemorySorter
UnsafeSorterSpillWriter
Storage
Storage
BlockManager
BlockManagerId
BlockManagerInfo
BlockManagerMaster
BlockManagerMasterEndpoint
BlockManagerSlaveEndpoint
BlockId
BlockDataManager
DiskStore
DiskBlockManager
MemoryStore
BlockEvictionHandler
BlockData
BlockInfoManager
BlockInfo
DiskBlockObjectWriter
BlockManagerSource
ShuffleMetricsSource
ShuffleClient
BlockTransferService
NettyBlockTransferService
NettyBlockRpcServer
ExternalShuffleClient
OneForOneBlockFetcher
ShuffleBlockFetcherIterator
RDDInfo
StorageLevel
StorageStatus
BlockStoreClient
BlockStoreClient
BlockStoreClient
ExternalBlockStoreClient
StorageUtils
BlockReplicationPolicy
RandomBlockReplicationPolicy
Serialization
Serialization
SerializerManager
Serializer
SerializerInstance
SerializationStream
DeserializationStream
Executor
Executor
Executor
TaskRunner
ExecutorSource
ExecutorBackend
CoarseGrainedExecutorBackend
TaskMetrics
ShuffleWriteMetrics
ShuffleReadMetrics
ExecutorLogUrlHandler
ExecutorMetricsPoller
Broadcast Variables
Broadcast Variables
Broadcast
Accumulators
Accumulators
AccumulatorV2
AccumulatorContext
InternalAccumulator
AccumulatorSource
AccumulableInfo
Barrier Execution Mode
Barrier Execution Mode
barrier-execution-mode
RDDBarrier
Speculative Execution of Tasks
Deployment Architecture
Deployment Architecture
Architecture
Driver
Master
Workers
Internal IO
Internal IO
SparkHadoopWriter
HadoopWriteConfigUtil
FileCommitProtocol
HadoopMapReduceCommitProtocol
HadoopMapRedCommitProtocol
HadoopMapReduceWriteConfigUtil
HadoopMapRedWriteConfigUtil
Stage-Level Scheduling
Stage-Level Scheduling
ResourceProfile
ResourceProfileBuilder
Misc
Misc
Deployment Environments
Building from Sources
KVStoreView
InterruptibleIterator
Deploy Mode
HeartbeatReceiver
Data Locality
CompressionCodec
LiveEntityHelpers
LiveTask
LiveStage
Utils
Spark Tips and Tricks
Spark Tips and Tricks
Spark Tips and Tricks
Access private members in Scala in Spark shell
Task not serializable Exception
Running Spark Applications on Windows
Spark Local
Spark Local
LocalSchedulerBackend
LocalEndpoint
LauncherBackend
Spark Standalone
Spark Standalone
Configuration Properties
Master
Worker
ExecutorRunner
StandaloneRestServer
StandaloneAppClient
ClientApp
LocalSparkCluster
Monitoring
Monitoring
Logging
ListenerBus
SparkListenerBus
SparkListenerEvent
SparkListener
SparkListenerInterface
SpillListener
StatsReportListener
AsyncEventQueue
spark-debugging
ConsoleProgressBar
Spark History Server
Spark History Server
Configuration Properties
HistoryServer
HistoryAppStatusStore
HistoryServerDiskManager
EventLoggingListener
SQLHistoryListener
ApplicationHistoryProvider
FsHistoryProvider
HistoryServerArguments
ApplicationCacheOperations
ApplicationCache
ReplayListenerBus
EventLogFileWriter
JsonProtocol
Status REST API
Status REST API
ApiRootResource
ApplicationListResource
OneApplicationResource
StagesResource
OneApplicationAttemptResource
AbstractApplicationResource
BaseAppResource
ApiRequestContext
UIRoot
UIRootFromServletContext
Plugin Framework
Plugin Framework
PluginContainer
DriverPluginContainer
ExecutorPluginContainer
SparkPlugin
DriverPlugin
ExecutorPlugin
Tools
Tools
spark-submit
spark-submit
spark-submit
SparkSubmit
SparkSubmitOperation
SparkSubmitUtils
SparkApplication
JavaMainApplication
spark-shell
spark-class
SparkLauncher
Internals
Internals
SparkSubmitArguments
DependencyUtils
SparkSubmitOptionParser
SparkSubmitCommandBuilder
Main
AbstractCommandBuilder
RDD
RDD
RDD
Custom RDDs
Custom RDDs
CoGroupedRDD
HadoopRDD
MapPartitionsRDD
NewHadoopRDD
ParallelCollectionRDD
CheckpointRDD
ReliableCheckpointRDD
ShuffledRDD
SubtractedRDD
Operators
Operators
Operators
Transformations
OrderedRDDFunctions
PairRDDFunctions
AsyncRDDActions
Actions
Partitioners
Partitioners
Partitioner
HashPartitioner
RangePartitioner
RDD Lineage
Caching and Persistence
Partitions and Partitioning
Partition
RDD Checkpointing
RDDCheckpointData
LocalRDDCheckpointData
ReliableRDDCheckpointData
Shuffling
Dependencies
NarrowDependency
ShuffleDependency
Aggregator
Metrics
Metrics
Configuration Properties
MetricsSystem
MetricsConfig
Source
Sink
Sources
Sources
JvmSource
Sinks
Sinks
MetricsServlet
Demos
Demos
DiskBlockManager and Block Data
Exercises
Exercises
One-liners using PairRDDFunctions
Learning Jobs and Partitions Using take Action
Spark Standalone - Using ZooKeeper for High-Availability of Master
Spark's Hello World using Spark shell and Scala
WordCount using Spark shell
Your first complete Spark application (using Scala and sbt)
Using Spark SQL to update data in Hive using ORC files
Developing Custom SparkListener to monitor DAGScheduler in Scala
Developing Custom RDD
Working with Datasets from JDBC Data Sources (and PostgreSQL)
Causing Stage to Fail
Web UI
Web UI
Configuration Properties
WebUI
WebUIPage
WebUITab
SparkUI
SparkUITab
Jobs Tab
Jobs Tab
Jobs
JobsTab
AllJobsPage
JobPage
Stages Tab
Stages Tab
Stages
StagesTab
AllStagesPage
StagePage
PoolPage
Storage Tab
Storage Tab
Storage
StorageTab
StoragePage
RDDPage
Environment Tab
Environment Tab
Environment
EnvironmentTab
EnvironmentPage
Executors
Executors
Executors
ExecutorsTab
ExecutorsPage
ExecutorThreadDumpPage
BlockStatusListener
EnvironmentListener
ExecutorsListener
JobProgressListener
StorageStatusListener
StorageListener
RDDOperationGraphListener
RDDStorageInfo
LiveEntity
LiveRDD
UIUtils
JettyUtils
CompressedMapStatus
¶
CompressedMapStatus
is...FIXME
Last update: 2020-11-24