Reading notes from 2017; technical explanations revised November 17, 2026. Architecture and scale observations refer to the source period.

An orchestrator schedules containers

Kubernetes manages workloads; it is not a Docker-like box. A pod is a scheduling unit containing one or more containers. For a Spark application, the driver coordinates work and executor pods run tasks. Kubernetes schedules those pods onto nodes.

The 2017 presentation predates later Spark/Kubernetes releases. The outline remains useful, but deployment flags and requirements must come from the Spark version in use.

An illustrative submission

For Spark 3.5.3, a cluster-mode submission has this shape. This is an environment-specific command template: replace the API server and image with values from your test cluster. The image must contain the matching Spark distribution.

spark-submit   --master k8s://https://KUBERNETES_API:6443   --deploy-mode cluster   --name spark-pi   --class org.apache.spark.examples.SparkPi   --conf spark.kubernetes.container.image=YOUR_REGISTRY/spark:3.5.3   --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark   local:///opt/spark/examples/jars/spark-examples_2.12-3.5.3.jar 10

The spark service account needs permission to create and manage executor pods in its namespace. Resource requests influence scheduling; limits and memory overhead affect whether the application survives pressure. Capture driver logs and application status before deleting test resources.

Follow the lifecycle

Submission creates the driver; the driver requests executors; executors run tasks and report progress. A completed driver is not the same as permanently retained logs. Decide separately where event logs and results live and how failed jobs are retried.

Running Spark on Kubernetes

Conceptual illustration; labels describe responsibilities, not a deployment specification.

Source slide locating Spark work within Kubernetes nodes and pods

Source slide locating Spark work within Kubernetes nodes and pods. Historical source figure; attribution and context are in References.

References