I don't see anything stopping the models from getting to that point, though. So far we've gone from the models not being able to do math at all, to improving on progressively more difficult problems, to claims of "it's just a counterexample", to now producing constructive proofs and new mathematics on the hardest problems we currently have. All within less than two years.

Being able to do a task well and explain what you're doing in an organized way are fairly orthogonal, even in humans. I don't see why the skills the models currently have wouldn't transfer, even if the results look psychedelic. The next step would obviously be to improve trust by improving the way in which results are communicated.