Re: GPT5, are there any .. reasonable/credible sources of information on the subject? I've become deaf from all the speculation and while i am very curious, i'm unsure if anything substantiated has actually come out. Especially when considering speculation from Sam himself.
Moreover, previous advances in GPTs have come from data scaling: throwing more data at the training process. But data corpus sizes have started to peak - there is only so much high quality data in the world, and model sizes have reached the limits of what is sensible at inference time.
What OpenAI can do while they are waiting is more of the easy stuff, for example more multimodality: integrating DALL-e with GPT-4, adding audio support, etc. They can also optimize the model to make it run faster.
Though my guess is that GPT-5 will be the last model which gains significantly from just adding more compute to the current transformer architecture.
Who the hell knows what comes next though?
In any case this wouldn’t be a data size scale but a model size scale. Model size scaling has its own set of limits, such as the aforementioned inference times. It seems safe to assume that OpenAI chose data and model size values for GPT-4 that were basically as big as they could go.
We don’t know anything about GPT-4 or how it is trained or what the training data is, so from that perspective anything we can say about it is unfounded speculation. This is an important asterisk to any discussion of it.