Hacker News Reader

Comment by peterwwillis | Hacker News Reader

These paradigms get thought up by people who don't run big systems, and then generalized to a much larger degree than they are actually applicable.

Segregating compute from storage is untenable for many kinds of workloads. Not to mention a single pipe to a single shared storage is a huge SPOF.

The best model for random devs who just want to scale up their jobs is to model what the dev is using for dev. They usually use a laptop - fast local disk, plenty of memory, fast CPU with speed bursts, fast interconnects, and no network latency or interruption. Build your app for that, and then spawn a new app for each compute node, and scale by nodes, and don't depend on the network for your app to survive. You may recognize this paradigm: it's what we were just starting to do 20 years ago. It's very resilient to failure, it isn't fancy or complicated, and it scales very well.

View on HN

Original paper author here. I think we are in vehement agreement!

1a. we don't run big systems. Well, I don't, at least -- my coauthors wrote spark and similar systems, so they're certainly aware of some of the challenges, but for us the major appeal here is someone else (cloud provider) is running a tremendous amount of the system!

1b. I think you're totally right that these sorts of systems tend to be generalized beyond their sweet spot. We have had quite passionate debates internally as to the amount of complexity we should add to the underlying system to support richer BSP efficiently. I just want to make sure the "obvious" stuff works as easily and quickly as possible (that is, basic map)

2. Segregating compute from storage _is_ untenable for a lot of workloads! But for a lot of the compute-heavy work we do, it's not. And we were really, truly shocked at how big the bandwidth to S3 really is -- for our compute applications, this degree of data disaggregation

3. "Model what the dev is using for dev" is _exactly_ what we're trying to accomplish. We even try and replicate the dev environment (thanks to anaconda, cloudpickle, etc.) as transparently as possible. That's exactly what we're trying to achieve, at the end of the day -- get back to where condor (another bird-named job system that we love) had us at 20 years ago.

No one runs big systems on such a platform until is has been used and tests by many parties. But it has to start somewhere!

I remember having many discussions about exact this kind of platform 20 years ago. Still waiting for it....

My question for you is how do you bill for usage? This would seems like just the kind of platform that could use an ICO.

Why didn't you write a nice python wrapper for AWS Batch? You'd get all the benefits of fully managed code, but none of the limits of Lambda functions. I don't see how any of your workloads are super latency sensitive -- a trivial AWS Batch job completes in ~2 minutes on a cold start.

I re-read the article and now it makes more sense. The takeaway seems to be "serialize your functions in bulk and de-serialize later in bulk": in essence, treat your compute like your storage. But I don't see how this is distributed computing for the 99%?

PyWren appears to be what some of the serverless platform providers were missing, and also repeats a bit of history from the bad old days of parallel processing. While it totally works for a lot of scientific and a few business cases, it doesn't work for the general case. It's like saying MPI was the best solution for the 99%. But I might be overlooking another detail.

The biggest problem with depending on one aspect of a system for performance is that it becomes the bottleneck. We'll rely on low-latency, high-bandwidth storage? Now if it becomes low latency or low bandwidth, our app is boned. We'll rely on loads of cheap compute cycles? If compute nodes start choking, our app is boned. We'll rely on RAM? Our data becomes bigger, and our app is boned. We'll just scale horizontally? Our network gets partitioned, and our app is boned.

It can work fine for systems that don't require performance guarantees. If that's the 99%, then this seems like a good solution, though it also seems like you're trading off learning a complex stack for learning HPC development models.

> These paradigms get thought up by people who don't run big systems

This is explicitly for people that don't run big systems – "the 99%".

> The best model for random devs who just want to scale up their jobs is to model what the dev is using for dev.

That doesn't seem like it will work if "random devs" aren't writing their code to even be capable of running on multiple nodes. You're also ignoring points made in the post:

> Many more casual users of data processing platforms (the 99% from the paper title) don’t want to spend hours understanding a complex stack of e.g. Spark, Scala, HDFS, Yarn and the JVM. Even the choice of instance types (70 on AWS at the time the paper was written), cluster sizes, regions and so on can be bewildering.

AFAIK, there's nothing that will automatically pick a stack, instance type, etc. based on a random dev's laptop (or desktop).

It is for people who run big systems because the title includes the words "distributed computing". They might as well have called this paper "building skyscrapers in your back yard" (which is a pretty good name for a paper I think!). If you don't write your app to be capable of running on multiple nodes, you aren't doing distributed computing - you're doing concurrent computing.

If there's nothing out there that picks a stack for you, that could certainly be automated based on a Q&A for the dev. But it's arguable that this decision should not be automated, because this decision directly impacts how the app works.

Some banks out there have been using systems very similar to this (in fact using cloudpickle too iirc), in production, and at scale, for years now.

Reply on news.ycombinator.com