Hacker News Reader

Comment by freeone3000 | Hacker News Reader

Will it? Or, like capital-intensive industries of the past, will deep learning funnel its profits into bigger and bigger computers, as has been done in the past and will be done again?

View on HN

These are order-of-magnitude increases. If it costs $5m to train GPT-3, which is 100x more compute than GPT-2, then it may cost $500m to train GPT-4, and $50b to train GPT-5. This is what is meant by economically (not to mention environmentally) unsustainable.

Environment aside (I don't even think the CURRENT rate of training is environmentally safe) -- I see no inherent problem with a 100x increase in cost per step holding anybody back. Once it's trained, you can run it much cheaper. Who's to say $50 billion of value can't be extracted from GPT-5?

> Who's to say $50 billion of value can't be extracted from GPT-5?

Perhaps it can. (Though the number of companies that can afford to train it is rather small.) But can $5 trillion be extracted from GPT-6? Even if it can, who can afford to train it?

If GPT-5 is capable of creating a cost-efficient implementation of GPT-6, then maybe we don't need to worry about the extrapolated $5 trillion price tag.

Using electricity is not inherently damaging to the environment. Very low cost and high power zero-carbon generation sources exist (hydro and nuclear). For scale also keep in mind that all the datacenters in the world still use much less power than is used for smelting aluminium.

Also, processors turn electricity into heat - if you can find a useful way of harnessing that heat, the computation is practically free. Deep learning is fairly latency-insensitive and is almost never mission-critical, so it's not inconceivable to imagine that the boiler rooms of apartment buildings and office blocks could become mini-DCs.

Also, once there are billions of value to be extracted, specialized chips will appear just to train those models (e.g. wafer scale analog processors, could easily be 3 orders of magnitude more efficient than GPUs/TPUs).

oh wait, decentralized, publicly controlled infra. What was the cloud about exactly?

Reply on news.ycombinator.com