Spark: Lazy Evaluation & the DAG

Transformations only build a plan. An action makes Spark run it.

Driver · your programempty plan
Transformations lazy — they only add a step to the plan
Caching lazy too
Actions trigger a job
Result on the driver
No action called yet — so nothing has been computed.
Pace
Cluster · the plan (DAG) over 3 partitions
narrow step: data stays in its partition wide step: data is shuffled not computed cached in memory