With models, there’s no reason that a model error in company A can’t be fixed for all of company A, and companies B-ZZZ.
- The mistakes made aren't "model errors" typically; you can't point to some aspect of a model and say that was at fault.
- You can't submit a bug report to a model provider for a mistake made when using a model, and you can't* submit training data to be incorporated in the next release of the model.
- If you own your model and are training it yourself, other companies won't see a benefit.
- You probably need to fine-tune models for each specific role and context so you don't just diffuse all the learning; lessons learned won't be applied to all your junior dev models, but you don't want them all to learn something specific about product A.
- If you take this to its logical conclusion you will invent a new role of "model manager" and associated hierarchy to ensure that training is effective and timely, and that company-wide lessons are applied across the model fleet.
- This is all impractically expensive.
If it were practical to have LLMs learn as they go, that would be a bit of a shake-up, in much the same way that a house fire is a bit of a warm up.
* Well, everything you submit to a model provider is likely winding up in training data anyway, no matter what your contract says.
and getting an improvement to some random unrelated 3rd party give us...?
maybe -- and it's a big maybe -- their improvements could help us to. but that's not a given.