Hacker News Reader

if you’re at cluster scale, i would skip right over dask and go to spark

or hadoop

I'll also suggest inscribing stone tablets with a chunked version of your computation and distributing the described chunked computation to your friends, and then gathering the tablets and finishing up the compute.

It's scalable to ~billions of nodes.

but that exists already and it's called databricks

Reply on news.ycombinator.com