Slides

https://github.com/keypointt/reading/blob/master/spark/2017_spark_spring_SF_Google_Kubernetes.pdf

note

Concept of Nodes and Pods

Kubernetes is a docker-like, self-contained box. Request process steps:

1. Spark Submit submits job to K8s
2. K8s schedules the driver for job
3. Driver requests executors as needed
4. Executors scheduled and created
5. Executors run tasks
6. Driver “completes” job and persists logs

Reference:

• Kubernetes Community: https://github.com/kubernetes/community/
• https://spark-summit.org/2017/events/apache-spark-on-kubernetes/