LLM agents play Pokemon showdown | Hacker News Reader

LLM agents play Pokemon showdown

olivergrabner.com

31 pointsgrabnero17 comments

As a tangent, there has recently been drama with AI and Pokemon Showdown because someone trained a bot (using standard reinforcement learning + a rewrite of the Pokemon battle engine to Rust for max self-play parallelism) that is apparently good enough to get to the top of the leaderboard.

Yeah I saw that too which was partly the inspo for this experiment. I feel like its kind of getting to a place like in chess where AI is getting better and while theres alot of drama right now, I feel like it'll be something like stockfish in the future

The difference is that many believe that the kind of bot detection algorithms used by chess.com probably can't be made for PS, which means there's no real way to give average players a bot-free ladder experience. It also basically invalidates Smogon's suspect test system for rule updates, which assumes that players who can reach an ELO cutoff have reasonable views on how to improve a format.

There's online tournament cheating potential as well, but from my time in that community it was pretty common knowledge that ghosting by top level human players was just accepted if you were in the right circles.

Oh, I haven't been keeping up with Showdown news lately... which format? I assume either OU or randbats?

That could be good news.

We may are approaching the point that online games become just playing against bots. I would love to see arcades and gaming cafes back. Real human connection instead of playing against bots. Ironically. For many years. The internet was the place to play against humans. AI, decompilation, and other tech may turn that around.

If the rise of multiplayer games didn't encourage LAN gameplay why would removing the need for a second player bring people to play in person

Did you try out different reasoning levels? Wondering if messaging is basically a proxy for additional reasoning

> Pokémon doubles is a game-theory environment without full information. The opponents' sets are hidden[...]

I'd be curious to see how a model fares in VGC rules with open team sheets, since that's where you swap out information-hiding games for good old-fashioned mind games :P

> [...] and damage is random [...]

Not to nitpick, but since damage isn't truly random, did it take possible damage rolls into effect? Being able to tell whether your opponent has a damage-boosting item/ability matters a lot!

Pokémon Showdown isn't a game so much as it is competitive Wiki reading.

Not going to lie I agree lol but still interesting to see

what was the selection for the teams? random per round? how does it play against people in ladder?

Yeah! So I did teams from a fixed, random seeded pool and then would rerun it with the switched teams. It also doesn't play against ladder yet but I played against it and it's not horrible. It plays better than an amateur but I was able to beat it a good amount

That makes sense -- these are details that would be good to have in the methodology :)

Would be interested on the results from running a similar experiment with more powerful models, though that's more of a curiosity thing than anything lol

Jev will give better results on this kind of thing.

For sure there are many better models for this use case[0]

At the same time, it’s interesting seeing how generally capable language models/LLMs/agents/harnesses are getting

We’ll probably see a lot more experiments and benchmarks like these in the near future

0: simple classifiers can play quite a few game, including Doom https://playground.jeffyclassify.com/#doom

Yeah Jev can play it but Jev cant communicate with its teammate in the 2v2 + it cant reason and is just a decision model

I agree with you

> but Jev cant communicate

However, it can chat https://news.ycombinator.com/item?id=49778162

Not very well of course :)

Reply on news.ycombinator.com