Another way to look at Dataflow pricing as it related to this type of technology. Let's just assume for a second Dataflow and other similar technologies execute your pipeline on your data within the same resource-time amount.
Dataflow lets you set an upper bound on these resources, and lets you auto-scale, with essentially per-minute granularity. Alternatively, you can create a cluster of fixed resource amount. Cost of Dataflow is $0.01 per hour per CPU, in addition to those resources.
Dataflow's model hence either guarantees 100% resource efficiency, or you specifically opt into the upper bound of spend. A deployment model that lets you "spin up a cluster and pay for it" is intrinsically less efficient.
I wrote a blog on this WRT BigQuery, but Dataflow is right there [0].
[0] https://cloud.google.com/blog/big-data/2016/02/understanding...
(Work on Google Cloud)