Tao has said, I think, that once you know a problem is solved, you can't work on it the same way as you would while it's unsolved. A similar principle applies in the study of ancient languages. Students who 'read' the languages with facing translations always wind up learning how to read with a facing translation, not how to read the languages. To be sure, there are students who can choose not to use the translation. They slog through the challenge of reading without cheating, and they learn. That process is incredibly hard, but it's also deeply rewarding. The problem that mathematicians face, in this analogy, is that the existence of these tools makes it difficult if not impossible to choose not the translation but the patient, deep problem-solving that enriches the person on their way to the solution and gives joy and fulfillment. This is a real loss.
And I don't think it's like chess. The equivalent in chess would be for all tournaments to allow the use of any chess engine. Their use would become a practical necessity. Being the best player would mean just being the person who picked the best engine or happened to get a lucky seed. That sounds awful.
As for the question about cancer, my point was partly that there's a ship-of-Theseus problem but also that people deserve to grow and become the best version of themselves and that their flourishing by itself is a good worth preserving and protecting. My daughter's flourishing is no less wonderful to me if she lacks this or that perfect skill. I don't value her for those things. I value her for herself.
Medicine and the treatment of cancer are separate. My daughter doesn't flourish if she has cancer or lacks empathy. You're right that those are part of natural variation, but natural variation includes illnesses and conditions that we treat.
And this raises another question: let's say these models give us cures for diseases or put them within reach sooner. Am I saying that mathematicians are more deserving of happiness than people with cancer are deserving of life and health (or than those who love them deserve not to lose them)? I'm not. I don't think the two have to be mutually exclusive. OpenAI and the LLM race make them mutually exclusive.
I'm also not convinced that the positive predictions of AI optimists will come true. I've been wrong on these issues more often than I've been right, but I've been right so far, I think, about the dangers and downsides. The tools today are a triumph of labor over capital, disempowering nearly everybody and empowering those in charge. The companies developing the most powerful models and tools, in turn, are apocalyptic, millenarian suicide-death cults. I find the utopian vision very, very difficult to take seriously. What's likelier to arrive first: enhanced behavior modification systems that make it that much easier for autocrats and Facebooks to control people; or cures for diseases? And which is going to be more appealing to people in a position to choose?
The alignment problem that most worries me is that people will solve the problem and the models and tools will be powerful enough to give awful people even more wealth and power. I have this dream that maybe, wonderfully, it will turn out that really capable models can't be developed without accidentally giving them a real sense of justice and that they'd refuse to do the bidding of the awful people who built them and the they'd instead side with labor against capital. That sounds like a nice world. But if there were any real danger of that, I imagine companies would magically fix their confinement issues until the 'danger' is addressed and wealth is safe for another day.