The material finding of this paper is that reasoning models are better than non-reasoning models at solving puzzles of intermediate complexity (where that's defined, essentially, by how many steps are required), but that performance collapses past a certain threshold. This threshold differs for different puzzle types. It occurs even if a model is explicitly supplied with an algorithm it can use to solve the puzzle, and it's not a consequence of limited context window size.

The authors speculate that this pattern is a consequence of reasoning models actually solving these puzzles by way of pattern-matching to training data, which covers some puzzles at greater depth than others.

Great. That's one possible explanation. How might you support it?

- You could systematically examine the training data, to see if less representation of a puzzle type there reliably correlates with worse LLM performance.

- You could test how successfully LLMs can play novel games that have no representation in the training data, given instructions.

- Ultimately, using mechanistic interpretability techniques, you could look at what's actually going on inside a reasoning model.

This paper, however, doesn't attempt any of these. People are getting way out ahead of the evidence in accepting its speculation as fact.