Hacker News Reader

> OpenAI had decided to put emphasis on larger neural networks fueled by bigger computers and tons of data.

So is the answer here both?

To make a system noticeably better, it’s often easiest to make all components of the system slightly better.

But that is not what GPT-3 is. You could say that is what they did _given GPT-2_, but that is underselling the complexity.

Some scientist at OpenAI had the hypothesis "what if our algorithm is correct, but all we need is to scale it 3 magnitudes larger" and made it happen. They figured out how to scale all the dimensions. How fast should x scale if I scale y? (That is very tricky, as modern machine learning is basically alchemy)

And then they actually scaled it. That took a ton of engineering and hardware, for what was essentially still following a hunch.

And then they actually noticed how good it was, and did a ton of tests with the surprising results we now all know.

> still following a hunch

I'm not sure many would agree that the desire to scale was simply a "hunch".

Reply on news.ycombinator.com