I wonder whether they pulled their article because they changed their mind. Maybe over time they got to understand the difference in use-cases and tradeoffs. Though it's definitely true that plain MapReduce can be shockingly inefficient, systems like Spark which add rich data structures and in-memory caching make a big improvement towards closing that gap. But it's still not a panacea, and I've definitely seen usecases where SQLite could replace and far outperform Spark, but the general programming model behind Spark is still a powerful and widely applicable abstraction.

My main complaint with Spark is that it's pretty hard for non-experts to debug crashes and failures. While the programming model beautifully abstracts away all the complexities of parallel and distributed programming with its headaches of managing clusters and dealing with synchronization and communication, the entire abstraction bursts instantly when debugging is needed. Suddenly you need to understand the entire Spark programming model from top-to-bottom, from DataFrames to RDDs and partitions, along with implementation details like shuffle files, DAGScheduler, block caching, Python-JVM interop, OOM killer, and on and on.

And there's absolutely no incentive for the commercial vendors to improve this situation in the open source project. (This isn't specific to Spark though, the entire Hadoop ecosystem seems to operate on hidden complexity in the open source version with vendors who make money by providing support and more usable distributions.)