Hacker News Reader

A chess scoresheet sometimes contains mistakes but chess players can figure out in many cases what was meant by thinking of what moves make sense and considering the level of play so far. Popular AIs tools fail at that.

View on HN

Chess is an interesting case. I remember in 2023, GPT 3.5 or something used to be surprisingly good at chess. There was even a "stochastic parrot chess" website [1]. I recall it was playing decently at around a 1800 level. Even as a fairly okay player myself (2100 bullet on lichess), I struggled to beat it. However, modern LLMs are a lot worse at chess. I guess having too much chess data in the training set probably regressed performance on stuff that actually matters, like coding.

[1] parrotchess.com, no longer available. Previous discussions: https://hn.algolia.com/?q=parrotchess.com

> I guess having too much chess data in the training set probably regressed performance

I think the theory is that the LLM having a high chess ELO was a pet project of a researcher that left.

I feel like the most parsimonious example, giving how exceptional the results were, is that someone was simply cheating (e.g. hidden tool use).

> example

explanation*. I must have been quite caffeine-deprived when I typed that.

Reply on news.ycombinator.com