Spark: Lazy Evaluation & the DAG
Transformations only build a plan. An action makes Spark run it.
Apache Spark processes large datasets by splitting them into partitions and working on all of them in parallel. But Spark is lazy: transformations like map, filter or reduceByKey don't compute anything — they only add a step to a plan, called the DAG (or lineage). Work only happens when an action such as count() or collect() asks for a result.
New to Spark? Start the guided tour: it builds a word count in four short steps and then runs it, pausing at every moment of the job to show what is happening and where to look.
Then explore on your own. Notice which steps are narrow (data stays in its partition) and which are wide (data must be shuffled), how shuffles split the job into stages, how a second action can skip stages, and what changes when you cache() an intermediate result. Hover any dotted term for a definition.