The bad cancer example is interesting to the extent it reveals that one of the world's top mathematicians is apparently unaware that applied science runs almost entirely on half-understood semi-empirical methods. That's most true for medicine, given the extreme complexity of human life; but if you look at a modern SPICE model for a transistor, a device that we claim to understand at the level of subatomic particles, then you'll find it's full of curve fits. As an engineer, I find that perfectly normal. My job is to use all the tools at my disposal to meet some human desire (to not die of exposure, for a slightly thinner phone, etc.), and whatever fundamental understanding I might have is only one tool to that end.

My impression is that since pure mathematics doesn't attempt to meet those human desires they need some other objective, and that objective is human understanding. The loss of that is thus felt more heavily than in other fields. We could say it's their problem, and they need to get over it just like the chess players did; but the outside implications are broader here, since mathematicians working in fields they themselves considered useless have so frequently been wrong--in Hardy's Mathematician's Apology, he gave number theory as an example of such a field, unaware of what the cryptographers would achieve just decades later.

It's possible that AI-generated pure math will continue this trend of delivering extraordinary unexpected societal value. It's also possible that the humans won't ever sufficiently understand that math, and the machines won't ever sufficiently understand human desires, and that connection won't be made. I've never met a pure mathematician who considered those downstream applications to be an important contributor to their motivations; but as AI-generated math contributes to the argument to allocate a large and increasing share of GDP to datacenter buildouts, that question of whether downstream value requires human understanding seems pressing.