More specifically, for DOTA, they could track the progress and make sure there wasn't important regressions.. but this seems so general, how can they make sure it makes everyone' use-cases better?
It’s a fact of life. A different model will generate different outputs for the same prompts. And some of those outputs will be worse than they were.
But, if you use the same prompt with the same model, the output will always be exactly the same (content filters notwithstanding).
Isn't this only true if you set the temperature parameter in a way that renders the model deterministic?
If you ping the OpenAI API without any explicit model specification, it'll return davinci:2020-05-03, which has the version.