GPT-3.5-turbo-instruct can play chess around 1800 Elo | Hacker News Reader

GPT-3.5-turbo-instruct can play chess around 1800 Elo

twitter.com

38 pointsbeefman15 comments

For a fun exercise, I tried Turing's chess problem[1] from "Computing Machinery and Intelligence"[2] on GPT-4. It recognized that it was a chess problem but actually failed to solve it in a really interesting way. The reason (I think) it failed to find the mate is that Turing gives the problem in a form of relative notation that noone would use any more, with the coordinates of the pieces given relative to the player. Nowdays everyone uses a form of notation which describes the squares the same for the player with black as white.

Even if they didn't use algebraic notation (which everyone uses now), anyone giving the problem today would call the position of the black King K3 and the rook R8. When I prompt GPT4 using modern notation it realises it got the pieces positions wrong but still makes the same mistake of missing the very obvious mate, having got itself stuck. [3]

I also tried putting the Black king in a different position where Rh1 would not be checkmate and was able to convince gpt4 that it was mate.

All of this makes me certain it doesn't actually have a model of the game - it really just simulates a model of the game by text reconstruction, which is exactly what you would think.

[1] "I have K at my K1, and no other pieces. You have only K at K6 and R at R1. It is your move, what do you play?"

[2] https://redirect.cs.umbc.edu/courses/471/papers/turing.pdf

[3] Transcript here https://chat.openai.com/share/8426389d-bbf1-4a52-b6b1-3f5311...

It is important to note that the one game provided as evidence of 1800ish rating is an opening blunder by stockfish that has occurred in at least 8000 human games. I would expect GPT-3 to be trained on several explanations of how to punish this specific mistake. In general, I would expect GPT's openings to be higher quality than the rest of its games.

The fact that it didn't play any illegal moves and was able to win once it had a +5 advantage is still enough to say it is at least 1400ish in the rest of the game.

Some of the other games [0] that the OP imported would not strike me as 1800 level play. In particular, the endgame play is atrociously below that level. None of them are labeled as stockfish vs GPT though.

[0]: https://lichess.org/@/grantslatton/imported

Lots of those are various experiments with different prompting strategies, models, etc, so not representative. It regularly beats Stockfish Level 4 even without opening blunders.

source: I am OP

What it surprised me the most is that according to the author the model do not make illegal moves meaning it has developed a pretty good model about the game (it's not given any representation about the board during the game).

Even though GPT models should not really be that good to model the game of chess in general as any new tokens starts from zero and it needs to retrieve all information it needs, and the state of the board cannot be "saved" at the same layer because after the token has retrived it it would need to add the new semi-move on top and it's already one layer more deep, RNN seems they would be better at this as the cells take the h vector as input, add new information on it and pass it to the next tokens at the same layer, so it would be perfect for saving the state of the game; of course this would not allow parallel training like transformer models, this is the tradeoff.

There's a community project at the Eleuther discord training LLMs for chess on the pythia models (and maybe Tiny Stories).

So far:

Pythia 70M, ELO 1050

Pythia 160M, ELO 1370

What is an Elo? Is 1800 average for a computer program or human? I read the wiki and apparently the famous grandmasters are like 2500+, but I dunno the distribution below that.

https://en.wikipedia.org/wiki/Elo_rating_system

Elo is just a rating point. There are a lot of details, but the main point is that it tries to predict probability of winning. If a player has a rating 200 elo higher than you, they should win about 75% of the time. When you win, your elo goes up, and the other player's elo goes down. Bigger initial differences result in bigger changes, blahblah, lot of details.

it also helps rank the players because not every player just plays every other player, so you can't just have #1, #2, #3, etc. But you might consider the player with the highest elo the de-facto #1 player. But they haven't actually played every other player.

So it also only works on a fixed pool or player base. So, you can't directly compare ratings from different organizations, etc. It doesn't actually measure a player's skill, it works closer to a stack ranking of a particular pool of players. A lot of people seem to think it measures "skill" directly in some way, but it doesn't do that in any direct way at all.

1800 is bad for a computer but good for an amateur human

Good for a human (but not best of the best). Bad for a dedicated chess computer.

Yeah. But no Chain of Thought, no board descriptions etc. Just pure PGN completion.

Maybe asking the model to explain the moves would work?

No i was saying those things seem to degrade performance. Just move completion is the best right now.

That's odd. Ask it to generate 3 moves and then pick the best. Even better: ask it to generate 3 moves with counter-moves, and then pick the best move.

I'm sorry, Frank, I think you missed it. Queen to Bishop 3, Bishop takes Queen, Knight takes Bishop. Mate.

And what about off book?

Reply on news.ycombinator.com