mrry

"Project Nessie: Transactional Catalog for Data Lakes with Git-like Semantics"

https://projectnessie.org/

This supports lightweight branches, and transactional commits and merges. I haven't used it—and it seems cool—but it also seems a little heavyweight to get cross-table transactions on top of these table formats (which would be my primary use case).

[Disclosure: I designed and wrote most of the docs for TensorFlow’s Dataset API.]

I’m sorry to hear that you’ve not had a pleasant experience with tf.data. One of the doc-related criticisms we’ve heard is that they aim for broad coverage, rather than being examples you can drop in to your project and run straight away. We’re trying to address that with more tutorials and blog posts, and it’d be great if you started a blog on that topic to help out the community!

If there are other areas where we could improve, I’d be delighted to hear suggestions (and accept PRs).

Yes. The prebuilt binaries can be used with GPUs that support CUDA 8.0 and have compute capability 3.5 or 5.2. The package is now in PyPI, so you can install the latest version with `pip install tensorflow-gpu`.

We've made good progress on single-machine performance and memory usage in the latest release, especially for convolutional models (such as adding support for NCHW data layouts), and we'll continue to aim for best-in-class performance in that setting.

The cool thing about distributed TensorFlow is that it supports efficient synchronous optimizers, so you can scale up the effective batch size by using multiple GPUs, to get increased throughput without losing accuracy.

Thanks for the shout out!

A lot of the differences between the systems arise from the implementation choice of how to do aggregation in Hadoop 2.4.0 and Spark 1.3. There's nothing inherent in the RDD model, for example, that says the aggregation has to be done eagerly at the mapper; nor in the MapReduce model that says it has to be done at the reducer. Either system could support the other aggregation mechanism, and the only challenge would be in choosing which one to use.

Some former colleagues wrote a nice paper about the performance trade-offs for different styles of distributed aggregation in DryadLINQ (a MapReduce-style system), and evaluated it at scale:

http://sigops.org/sosp/sosp09/papers/yu-sosp09.pdf

Malte Schwarzkopf is the first author of the Omega paper:

http://eurosys2013.tudos.org/wp-content/uploads/2013/paper/S...

...so not entirely blind.

It's interesting to compare this with the Facebook stack, drawn by the same author:

http://malteschwarzkopf.de/research/assets/facebook-stack.pd...

…and it would be doubly interesting to see the "related work" arrows between the different companies' infrastructure platforms. We'd need a 3D visualization for that though.

An interesting comparison point: a single core on a late-2014 MacBook Pro can achieve runtimes for the same graph that are within a factor of 4 for WCC (461 seconds for FlashGraph versus 1700 seconds for the laptop).

http://www.frankmcsherry.org/graph/scalability/cost/2015/02/... (previously on HN: https://news.ycombinator.com/item?id=9001618)

There are also results for PageRank on that graph, which make the difference more pronounced. FlashGraph runs PageRank in 2041 seconds (I'm assuming for 30 iterations, per Section 4 of the paper), whereas the laptop takes 46000 seconds for 20 iterations.

I think in its current form, COST simply provides a reality check: for some problems, a given system may be completely inappropriate when a simple single-machine implementation exists. Along with raising standards for distributed systems evaluation, I hope that it encourages people to innovate on the single-machine implementations: for example, the GAS idea in PowerGraph [1] is an antecedent of the Hilbert-curve implementation in Frank's first blog post, and it would have been great if the original GAS implementation were separable from the particular distributed system for which it was built.

I fantasize about this work inspiring a competition like the sort benchmark [2], but for single-threaded graph algorithms. Not only would this encourage people to develop usable single-threaded implementations, but it have the side effect of increasing the COST for all distributed implementations....

As for providing scalability break points, I think this work goes some of the way, but doesn't take into account the fact that the input data for a particular real-world computation might not reside in a conveniently-compressed format on a local SSD. Extracting the Common Crawl from an HDFS cluster just to replicate the blog post's experiments could take much longer than the computation itself; extracting the social graph from Facebook's Tao [3] to do offline processing might be similarly non-trivial. Academic experiments don't typically take this cost into account, but recent work like Musketeer [4] has started to consider it, and even offer ways to automate the system choice problem.

[1] https://www.usenix.org/conference/osdi12/technical-sessions/...

[2] http://www.sortbenchmark.org/

[3] https://www.usenix.org/conference/atc13/technical-sessions/p...

[4] http://www.cl.cam.ac.uk/research/srg/netos/camsas/musketeer/

I would wager that an appropriately-sized Spark, GraphLab, or Naiad cluster could outperform a laptop when running PageRank on this graph. Distributed graph processing research got stuck in a rut when the gold standards for evaluation were the Twitter crawl [1] and uk-2007-05 web graph [2]. Now that the Common Crawl has made a much bigger graph available, I hope that we'll see it used in more performance evaluations—with a COST baseline, of course!

I'd also wager that those Spark, GraphLab, or Naiad programs would be far from optimal, by at least an order of magnitude. My ideal would be to see someone attack the distributed graph processing problem with the same eye for systems performance issues that the TritonSort folks brought to the distributed sort problem [3] and MapReduce [4].

[1] http://law.di.unimi.it/webdata/twitter-2010/

[2] http://law.di.unimi.it/webdata/uk-2007-05/

[3] http://www.alexras.info/pdfs/tritonsort-tocs13.pdf

[4] http://www.alexras.info/pdfs/themis_socc12.pdf

This is absolutely true! However, I'm not aware of anybody doing this for the graph processing systems being discussed in the post. Nor am I sure what that would mean. Admittedly if something takes 100 times longer to run on multiple machines than it does on a single laptop (as in the previous COST post [1]) then you might care about fault tolerance, but this is a contrived problem if you could achieve the same results faster on yor laptop.

[1] http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...

Our point in the COST paper was neither that distributed computation systems are bad per se, nor that a laptop is a fine substitute for many of the real-world deployments of distributed systems.

What these posts rail against is a tendency (post-MapReduce/Hadoop) to motivate and evaluate distributed systems solely in terms of their batch performance. This is a particularly common problem in graph processing research, where 2008-era general-purpose systems (i.e. Hadoop) have terrible performance, and so provide a straw-man baseline that a reasonably competent grad student should have had no difficulty in outperforming.

"Does my new system outperform a single thread?" seems like a proper hypothesis to test for a distributed system that claims only a performance motivation. Instead our grad student finds that there is a long list of peer-reviewed systems to compare against, and doing that should be enough, right? It would be fine if those systems had been tested against the first hypothesis, but in most cases they have not, so the new system can be much worse than a single thread (for a small set of benchmarks) and this will go undetected. The COST work is a remedy for this issue.

To return to your original question, I do think that distributed systems are good for some things. First, the OP concerns running graph algorithms on genuinely big data: 128 billion edges, and 3.5 billion vertices. A distributed system should be able to outperform Frank's laptop on that graph, and hopefully this post will encourage developers of new systems to aim higher and evaluate their systems on this scale of data.

The second is that distributed systems offer far more than pure batch performance. For example, geo-distribution is critical to providing a good user experience in web properties; and serving interactive queries with low latency and high availability is a particular interest of mine. We touched on the first part in the Naiad paper [1], and I expect this will become especially important for inference in online learning systems [2]. Hopefully the COST work will force the authors of these systems to try harder in their implementation, and we will all benefit as a result.

[1] http://dl.acm.org/citation.cfm?id=2522738

[2] http://www.cidrdb.org/cidr2015/Papers/CIDR15_Paper19u.pdf