Also probably in the training data: https://www.quora.com/What-is-the-least-odd-prime-factor-of-...
It's a public AIME problem from 2019.
It's a public AIME problem from 2019.
There is an in-between measure of usefulness which is to take a problem you know is in the dataset and modify it to values not in the dataset on measure how often it is able to accurately adapt to the right values in its response directly. This is less a test of reasoning strength and more a test of whether or not a given model is more useful than searching its data set.
My intuition has slowly been building towards thinking of LLMs (and potentially all generative models) are essentially search algorithms