It's worth pointing out that this is a multi-trillion (probably) parameter model optimizing an 800 million parameter model. There is no indication that it can improve itself with this same methodology.
The parent's point still stands, I think. Even if the multi-trillion param model is MOE, each expert is still (assumedly) larger than the model being optimized.
It would be interesting to see future work that characterizes the relative model sizes (and considers MOE vs not-MOE) to see what relative scales this result holds across.
This would be earth shattering. That would mean we’re essentially at RSI.
Occams razor would say that it was not only built for this purpose, but possibly multiple generations of good at this.
A year ago LLMs seemed to hit a wall, and then all the sudden they started accelerating as quickly as ever.
No one can legally confirm or deny that it's happening... But it sure feels like it's happening.