Hacker News Reader

Comment by sandGorgon | Hacker News Reader

>Distributed Training: Gloo is now supported for distributed training jobs.

This is very interesting. Can someone talk about the roadmap of pytorch here ? It seems everyone is kinda rolling their own -

Pytorch has a very confusing distribution story

- OpenAI runs Pytorch on Kubernetes with handrolled MPI+SSH

- https://pytorch.org/tutorials/beginner/dist_overview.html

- https://pytorch.org/docs/stable/distributed.elastic.html

- https://pytorch.org/torchx/latest/

- https://www.kubeflow.org/docs/components/training/pytorch/

- Pytorch-Biggraph is specifically using torch.distributed with gloo (with an MPI backend).

So here's the question - if ur a 2 person startup that wants to do Pytorch distributed training using one of the cloud-managed EKS/AKS/GKE services... what should you use ?

View on HN

The pytorch lightning people have come up with grid.ai, I personally have obtained good results by using pytorch lightning plus slurm on HPC machines. If I were a startup, I would probably try to build my own small HPC cluster, since that is far more cost effective than renting.

so most early stage startups get tens of thousands of dollars of free AWS credits. https://aws.amazon.com/activate/ 100K if ur part of a university accelerator.

it is far far more efficient (as a proportion of time-to-market) to rent and build on top of services.

Kubernetes is where the wider ecosystem is. I dont like it ...but it is what it is.

So Grid.ai is something like AWS Sagemaker. I wanted to figure out what someone can use on a readymade kubernetes cluster.

During my PhD I burned through ~10^5 of compute hours on 4-8 V100 GPU, 128 Core, 1TB+ HPC nodes. Your 10k AWS credits run out really quickly compared to that. And as a startup you are far better of spending some money on a few beefy machines before renting them.

So this thread is a bit of a segue. Even if u rent/buy your machines...my question was "which framework"

I definitely don't want to dispute your calculations.. However, I'm still puzzled what to use.

I would agree with you if the case was some sort of web service in general, but a GPU server that is only used in-house to run some common pyTorch training jobs is very simple to buy and install and run. Those cloud credits are going to get eaten fast as its a few $/hour, assuming you can get them (those programs are only available if you get your financing from a participating incubator or investor). But yeah if you do get 100k for free from someone you should of course use them first :)

Check out Determined (https://github.com/determined-ai/determined). It supports deploying onto k8s and handles running horovod (and soon other dtrain backends), with most of the complexity abstracted behind a few configuration values. Also, it gives you stuff like experiment tracking / hp search (asha) /scheduling / profiling and etc.

Disclaimer: I work for Determined.

"- OpenAI runs Pytorch on Kubernetes with handrolled MPI+SSH"

Is that published somewhere?

Reply on news.ycombinator.com