Scott Says:
Comment #91 October 8th, 2026 at 12:34 am
Jason #88:
They threw 8,000+ problems at the wall and lesss than 400 stuck to it. I’m much more interested to see the transcripts from the 7,600+ that were discarded.
ROFL. Why didn’t anyone ever think to say to Andrew Wiles, “forget about Fermat’s Last Whatever, tell us about all the math problems you couldn’t solve”?
Anyway, my prediction in this post was exactly right. No matter what AI does, no matter what happens in external reality, cope springs eternal in the self-satisfied Internet commenter’s breast.
https://scottaaronson.blog/?p=10169#comment-2041180Amazing. I have never known a mathematician dismiss a quantitative inquiry with such a flippant attitude. Yes, in machine learning research we like to measure the performance of our systems in fractional quantities, for example "we gave this system 100 problems and it solved 10: that's an accuracy/solve rate/pass@1 of 10%". Nothing wrong with that and nothing to cope with.
400 (to be charitable) divided by 8000 is 0.05. 5% accuracy (again charitably) is ... not great. Remember this is the revolutionary approach that is going to transform mathematics and solve all problems currently unsolved by humans. Quite unlike the expectations placed upon oh, I don't know, Andrew Wiles, for example. AI is going to leave mathematicians out of work because it can solve all maths.
All of them.
5% at a time. Now who's being willfully blind?
P.S. What an ungracious, rude blog host.