I'm glad to see the paper specifically contrast iterator approaches (pull model) versus a "data-centric" approach (push model). This was indeed the issue I've seen with Apache Spark's RDD.

Imagine a common pipeline:

  - input data
  - process data
  - output data

Using iterators, the output logic loops over an abstract next() interface provided by the process logic, which in turn loops over another abstract next() interface provided by the input logic. Given a long enough chain of dependent processes, the CPU's cache will be obliterated.

If we were to write code in a regular programming language by hand, we would never use such an inverted model. Instead, we would directly input the data, process it, and then output it. That's exactly what the paper proposes; the authors supply a callback function at the major stages of query evaluation. They use this model in the context of writing a compiler to speed query execution.

It's worth noting that the Spark maintainers also realized that iterators have bad performance and that newer versions of Spark follow the compiler approach:

https://databricks.com/blog/2016/05/23/apache-spark-as-a-com...