Hacker News Reader

Comment by Retric | Hacker News Reader

LLM’s are terrible at following the rules of Chess when playing a game. Determinism isn’t helpful because valid moves are so dependent upon the total state of the board.

Programming state gets way more complicated, but code is generally written to help people understand it greatly simplifying the process. Feed in “cursed” code leveraging obscure language features in creative ways and LLM’s and most people fail.

View on HN

LLM's on their own might fail, but coding agents will succeed because they will be able to write scripts which simulate the states correctly.

You're right that it's worth considering in the same way as the letter counting tricks, since coding agents can get around that in the same way. But I would say the letter counting has the advantage that it at least is easy for humans.

I am just as interested in making the language hard for LLMs as for LLMs+coding agents though.

LLM's were terrible at chess last year. Today, they can't beat stockfish but they can play competitively with the top 2% of chess players. Next year, who knows.

https://chessbenchllm.onrender.com

Look at the “legal %” field on that chart. That’s utter trash even counting per move vs games without errors. Further they outperform vs bots who stick to main lines more than normal players, actual performance vs humans means both lower ELO and more errors.

Note: * = Anchor player (fixed rating, used for calibration)

Reply on news.ycombinator.com