Hacker News Reader

SETI@home style distributed computation is not suitable for training something like GPT-3, unlike for SETI, the unit of work a node can do before needing to share it's output with the next node is really small, so very fast interconnect between the nodes is needed (Infiniband and NVLink is used in clusters to train it). It would probably take a decade to train such a model over regular internet.

View on HN

Maybe a case for a community colocation cloud where I a consumer can buy a system and colocate it in a large data center with great internal networking. Edit: typo

Handling heterogenous (and potentially untrustworthy) systems also adds overhead, not to mention that buying hardware in bulk is cheaper, so it makes the most sense just to raise the money and buy the hardware.

The problem is potentially solvable as generating solutions takes a lot of GPU time and verifying it is very fast. Aquiring input data may be a problem, but should be possible with dedicated models for this type of computation.

Are there any models/research optimised on working on this kind of small, distributed batches that would fit ie. ~10GB of commodity GPU?

Reply on news.ycombinator.com