These are famous games tested here though, and I tried to feed one problem to DeepSeek 4.1 Flash 1017 and it comes back at 66% confidence it is a Pac-Man (incorrect). That smells to me the problem setup heavily hint towards these 2D games. Might be resolution related to be honest.
I tried the same experiment with DeepSeek 4 Flash as well, on more than 4 games. And I think you might be right, the performance is much worse for other games:
https://news.ycombinator.com/submitted?id=wohoef
Great post. I looked into the reasoning trace, and it seems DeepSeek 4.1 Flash 1017 does know all the games you listed in the post, so there is no knowledge gap. Not sure about Qwen though. I will not be surprised if it doesn't know all these games.
To be honest I think the experiment would be insanely impressive even if you prompted explicitly that the game is one of 4 options.
From what I saw in the paper, no they did not. They tested that the specific change introduced increased performance on a single task (the one mentioned in the prompt). They did not study the impact of that change on overall model performance across a variety of benchmarks. But maybe I missed something.
But increasing performance on a single task was the point of the change?
His point is that he has read Shakespeare but not the linked article.
He can call on comments from the vasty deep, but shalt we find them relevant?