Hacker News Reader

Comment by zarzavat | Hacker News Reader

It’s unlikely to be any time soon. Despite productization by OpenAI, LLMs are still an active area of research. Research is unpredictable. It may take years to gather enough fundamental results to make a GPT-5 core model that is substantially better than GPT-4. Or a key idea could be discovered tomorrow.

Moreover, previous advances in GPTs have come from data scaling: throwing more data at the training process. But data corpus sizes have started to peak - there is only so much high quality data in the world, and model sizes have reached the limits of what is sensible at inference time.

What OpenAI can do while they are waiting is more of the easy stuff, for example more multimodality: integrating DALL-e with GPT-4, adding audio support, etc. They can also optimize the model to make it run faster.

View on HN

I keep keep hearing people claim we are at the end of corpus scaling, but that seems totally unfounded. Where has it been proven you can’t running the training set through in multiple epochs in randomized order? Who’s to say you can’t collect all the non-English corpus and have the performance transfer to English? Who’s to say you can’t run the whole damn thing backwards and still have it learn something?

When you train a NN, you train until the loss converges (or until you run out of money). If it happens that you exhaust your training data and the loss is still decreasing, then yes you keep going with a second epoch. On the other hand, if your loss has already converged then starting a new epoch on the same data is just burning money.

In any case this wouldn’t be a data size scale but a model size scale. Model size scaling has its own set of limits, such as the aforementioned inference times. It seems safe to assume that OpenAI chose data and model size values for GPT-4 that were basically as big as they could go.

We don’t know anything about GPT-4 or how it is trained or what the training data is, so from that perspective anything we can say about it is unfounded speculation. This is an important asterisk to any discussion of it.

You are missing the scaling on context size.

There are diminishing returns from compute time but it looks like even though they are diminishing there's still a fair bit on the table.

Though my guess is that GPT-5 will be the last model which gains significantly from just adding more compute to the current transformer architecture.

Who the hell knows what comes next though?

I know ChatGPT is trained on Wikipedia and other open datasets. I wonder what would happen if you gave it the entire Bing index?

Reply on news.ycombinator.com