Hacker News Reader

Comment by sailingparrot | Hacker News Reader

You are off by an order of magnitude at least. 256 TPUs-v4 (not pod), would cost you around 20k$/day. They actually used 512 TPUs (256 for base model + 128 for each of the two superresolution models).

Assuming an average training time of 1 week as you said, that gives us about 280k$.

It's also most likely trained for longer than a week, the base model for Dalle-2 was trained for 100-200k GPU hours, so between 2-4x longer than that, we can guess this is roughly similar.

You also never successfully train everything first try, so all in all, to replicate this work just from the paper, we are talking about at least 500k$.

View on HN

While that is what they did, they also used a batch size of 2048 while training. This is just to speed training up, not a hard requirement. It's easy for Google to justify more money on compute to save engineer iteration loops.

I'll have to read the paper for more details, but it would almost certainly cost less (and take longer) to train a model like this in a more resource constrained situation than Google faces .

Increasing batch size does not increase the cost of your training. The opposite actually: With bigger batch size (to an extent), models tend to converge slightly faster so you need less GPU hours. As for the rest, training for 1h with batch size of 2048 on 10 TPUs, or training for 8h with batch size of 256 on 1 TPU has the exact same cost, the cost is just spread over a longer time.

Couldn't a bunch of us shell out $5000~$50,000 and do this ourselves? Create a non-profit shell corporation outside US jurisdiction, issue shares, raise funds and open source the result?

The shares would simply be votes towards future training dataset endeavors as no profit would be booked here. Say you buy 5000 out of 500,000 shares, that would give you 1% voting power in what dataset to train.

TF Research Cloud. Especially if multiple people could use it with eg model checkpoints:

> 5 on-demand Cloud TPU v3 devices, 5 on-demand Cloud TPU v2 devices, and 100 preemptible Cloud TPU v2 devices for free for 30 days

So up to 7k hours on demand and 70k pre-emptible

Go right ahead, spend $5-50K of your own money and let us use it for free.

Who is going to be liable for the flood of child pornography that comes out of that setup?

Reply on news.ycombinator.com