> the world really has moved on to open models
Don't get me wrong: I'm all for open models, but I think it will get more and more difficult to distil-train them without (legitimate) access to frontier models.
Don't get me wrong: I'm all for open models, but I think it will get more and more difficult to distil-train them without (legitimate) access to frontier models.
People have no idea and everybody pretends to be an expert and ignore how good China is on AI research
If anything I bet these people will just use their knowledge to make even more money reselling tokens.
The set of models solving really most used/solved problems is a known, as opposed to the cases where it's unknown, which declines with usage over time.
If you want an Opus class model, it makes sense that you would train on what Opus outputs. But, if you want something better than Opus, training on the same data that Opus was trained on with the same architecture will only result in an Opus class model. Then, if your dataset also contains Opus outputs, many of which are wrong, then it makes sense that the model would have reduced performance.
All this to say that I don't think there's such a thing as a "Model Collapse," but there likely is a "Model Stagnation."