Exactly what I’ve found. Try giving an LLM any novel easy problem (IMO 2024 Problem 5 is a good example). And absolutely every llm out there fails miserably.
> AlphaProof, a new reinforcement-learning based system for formal math reasoning
> AlphaGeometry 2, an improved version of our geometry-solving system
"There are 7 cats in a playground and 61 plushies, ..." (insert basically the same problem, requiring the same solution)
Well... Then the LLM shall be able to solve it.
And many people will consider it's a novel problem and hence a resounding success.
I mean: it is a success, but not anywhere near as impressive as most think.
Now, given the token encoding I think naive letter counting is not something we should expect from LLMs, but still serves as a nice reminder to actually ensure the test/validation data is not part of the training data.