Reading notes from 2017; technical explanations revised November 17, 2026. Architecture and scale observations refer to the source period.
Memory is a budget with several consumers
A Spark executor needs memory for computation, cached data, object overhead, and runtime/native work. A large JVM heap is not the entire container budget. Python workers and native allocations can matter even when the heap itself is healthy.
Execution work such as joins and aggregation competes with cached storage according to the release’s memory-management rules. When work cannot remain in memory it may spill intermediate data to disk; eviction and recomputation can make caching less valuable than expected.
Follow a spill
Imagine a skewed aggregation where one partition receives most keys. The task’s working set grows, intermediate data spills, and its runtime becomes the tail of the stage. Giving every executor more memory may cost more than correcting the partition imbalance.
A useful investigation records task-level input, spill bytes, garbage-collection time, executor loss, and container memory. Distinguish an out-of-memory error in the JVM from a container termination or a Python worker failure.
Read the 2017 roadmap historically
SPARK-19489, SPARK-15689, and SPARK-15687 were follow-up pointers from the original talk, covering serialization, data-source APIs, and columnar execution. Their current implementation/status must be established from the chosen release. A roadmap slide is not a specification of today’s memory behavior.
Conceptual illustration; labels describe responsibilities, not a deployment specification.