This is less insightful then you might think, as it's not limited to modeling. We just have a tendency to use previous observations as truth for future endeavors. Mainly because it's usually fine and works fine.
You can see it in every second discussion about any topic that's currently being researched:
I.e. there is nothing guaranteeing we won't have a super viral virus with 90%+ chance of death. It could happen...
There is nothing guaranteeing that LLMs ability to output relevant data will get better, even if it's been tremendously improved within the last year alone.
You can basically see this in action whenever someone is making a prediction. The likelihood of it coming true might be good enough to work with it, but you can always have something go amiss, or a meteor out of currently unknown materials hits the sun causing a chain reaction which causes it to go supernova...