What these posts rail against is a tendency (post-MapReduce/Hadoop) to motivate and evaluate distributed systems solely in terms of their batch performance. This is a particularly common problem in graph processing research, where 2008-era general-purpose systems (i.e. Hadoop) have terrible performance, and so provide a straw-man baseline that a reasonably competent grad student should have had no difficulty in outperforming.

"Does my new system outperform a single thread?" seems like a proper hypothesis to test for a distributed system that claims only a performance motivation. Instead our grad student finds that there is a long list of peer-reviewed systems to compare against, and doing that should be enough, right? It would be fine if those systems had been tested against the first hypothesis, but in most cases they have not, so the new system can be much worse than a single thread (for a small set of benchmarks) and this will go undetected. The COST work is a remedy for this issue.

To return to your original question, I do think that distributed systems are good for some things. First, the OP concerns running graph algorithms on genuinely big data: 128 billion edges, and 3.5 billion vertices. A distributed system should be able to outperform Frank's laptop on that graph, and hopefully this post will encourage developers of new systems to aim higher and evaluate their systems on this scale of data.

The second is that distributed systems offer far more than pure batch performance. For example, geo-distribution is critical to providing a good user experience in web properties; and serving interactive queries with low latency and high availability is a particular interest of mine. We touched on the first part in the Naiad paper [1], and I expect this will become especially important for inference in online learning systems [2]. Hopefully the COST work will force the authors of these systems to try harder in their implementation, and we will all benefit as a result.

[1] http://dl.acm.org/citation.cfm?id=2522738

[2] http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper19u.pdf

So basically big systems only make sense if the computation time on a single system exceeds the transfer time off of that system to the remote nodes + the time for that data to work it's way up the local node memory hierarchy.

This can happen when the data is big and the computation has unavoidable serialism in each part of the computation. Otherwise you'r likely to be able to coax a single system to just do it faster and with far less complexity than working on a compute cluster.

(I'm ignoring big systems with fast shared memory, like classic super-computers, where there effectively is zero transfer time since each node uses the same memory pool)