Can anyone speculate on the best way to update an LLM with new data? Do you fine-tune it on just the new data? Do you mix it with the old data and train for another epoch?
But there would be a difference in outcome compared to training a model from scratch with all the info available in 2023. Training from scratch would ensure that newer data has equal footing with older data. So a model trained from scratch in 2023 might have a more balanced understanding of both old and new data. Also there's a chance that the model might identify new patterns or nuances.
When you just fine-tune the existing model there might be areas where the model's response leans more towards the knowledge from 2021.
Having said that, training the model from scratch with all the new data would be incredibly resource-intensive, so I'm pretty sure OpenAI didn't go that route.