It looks like climbing a tree to reach the moon to me.
AI like chatgpt aren’t writing new code, they’re outputting code following their inputs, which is code found on the internet. That’s useful to an individual, but by its nature can’t go beyond what’s already there.
If you hunt around on SO, you aren’t going to find code to generate an AI that can code a better AI. Because no one’s created that code. That’s how current AIs work too, so they aren’t going to be able to do it either, no matter how large and capable the models become.
In short, I don't think people are being imaginative enough about this stuff, given the number of additional ways in which it can be improved.
> Evolution through Large Models
They are using a diff-LM trained on GitHub commits to generate mutations to a population of problem solutions expressed as code.
https://arxiv.org/abs/2206.08896
The non intuitive bit is why are these models surpassing the original programming? It's because they got a secondary source for learning - the feedback they get from generating solutions and testing them. Learning through feedback, be it reinforcement, evolutionary or gradient based, is what can push AI forward.
AlphaZero started with no human knowledge by pure self-play and beat all humans in a few days of practice. It learned from game outcomes as a feedback source. AI can learn from many things, not just from humans. But if it doesn't require humans it usually requires massive computation, simulation or search, maybe even real world testing which can get expensive.