Show HN: Gemini 4 Argon plays StarCraft
starskirmish.com
I've refreshed StarSkirmish Bench, a benchmark where LLMs play StarCraft BW by writing code, with results from the flagship models of OpenAI, Anthropic, Google, and Muse
starskirmish.com
So it almost seems like a situation where you cannot really have your cake and eat it, at least not in the way it currently works.
Vals IOI, olympiad programming problems solved in C++, and WeirdML v3, unusual machine learning coding tasks also mirror StarSkirmish Bench.
I did a StarCraft: BW hillclimb experiment with GPT-6 Astra and Claude 5.5 Opus. Peter Gostev did a Chess hillclimb saw similar behavior as I did with GPT-6 Astra.
Anyone else feels like they can tell when the underlying model is MoE? Dense models are more concise in their answers. MoE seems to stall for time, writing vaugely until a right set of weights gets picked and commits to an answer. MoE also backtracks a whole lot more.
I think those kinds of questions would be more interesting than the raw win/loss ratios.
I did a 48 hour test with GPT-6 Astra and Claude Opus 5.5 and we saw some interesting uses of Psionic Storm.
I've open sourced some of GPT-Astra's work (below). It actually did an interesting amount of changes in strategy over time as it's opponents became more difficult.
Anecdotally, in game benchmarks I've run, OpenAI models seem particularly adept at finding the meta game and exploiting it...
The unsexy perspective on StarCraft is that it's a game about logistics. You need to keep making things, not get supply blocked, fit all your buildings, not get your units trapped, and then bring all your units to a fight at a good time and at the same time.
Astra and Opus both cleared this bar pretty quickly, both settling on one-base Dragoon + Observer play, taking expansions pretty late, on 3 Gateways. This strategy is fine against everything and great against nothing, letting you win against anyone that you can out-execute. StarSkirmish gives agents access to the FastApproximation combat simulation library, which makes better combat predictions than the B-tier bots do, and let their bots win by trading armies better against them.
But the A-tier bots (Locutus and BananaBrain) are already strong at execution, and Astra wasn't making progress against them. It switched to a proxy 2-gate Zealot rush. This worked well on a map pool with mostly 2-player maps (so you can place the Gateways very close to your opponent). But it's also an astute choice for a new bot because it doesn't require you to be good at anything in StarCraft that happens after the first four minutes!
It's also smart as a variance-maximizing strategy. In many games, if your opponent is superior, you want the game to be decided by as few events as possible. Because with a bad coin, you're more likely to win 1 of 2 flips than 10 of 20. Proxy 2-gate shortens the game and the number of interactions required to win.
It's also advantageous given the format, where you get unlimited tries to win a series against each tier's opponents. If you can make the games a coin flip, and then take more coin flips, you can clear a tier even with an overall poor winrate. I do think this aspect of StarSkirmish is worth addressing in the future.
Astra's proxy gates beat the A-tier bots, and it moved up to StarSkirmish's S-tier (Stardust and PurpleWave, though really I'd consider BananaBrain completely our peer in ability).
A brief aside about bot tournaments: The AIIDE conference and historically the COG tournaments had every bot play hundreds of games against one another, and ranked then on overall winrate. I have never liked this because it does not resemble human StarCraft (where players play a best-of-N series), unbalances the races (Zerg vs. Zerg is more volatile and winrates should be closer to 50%), and creates some twisted incentives. One of those incentives is ensuring you beat weaker opponents 100% of the time instead of merely 95%, which means defaulting to reliable-but-weaker strategies to beat weaker opponents, and having a large bag of available strategies to find something which exploits trickier opponents.
And in this iteration of StarSkirmish, where benchmark bots are mostly unable to store data between games, Stardust ran one basic conservative strategy almost every game. This weakens Stardust, but meant that Astra's proxy gates were unlikely to win. So Astra had to do something very uncomfortable for a coding agent: Go back to the drawing board and try something completely different.
Astra completely switched gears, landing on a strategy that PurpleWave also likes to use against Stardust: Use Dark Templar to secure an expansion. This is particularly effective against Stardust, who does not use Shuttles + Reavers, and thus does not gain utility from the Robotics Facility it requires to make Observers.
From there Astra progressed slowly, but discovered a shared weakness of PurpleWave and Stardust: Both of us are designed to dodge Psionic Storms, but that feature was broken in both our bots simultaneously (for unrelated reasons!).
So Astra's Dark Templar opening transitioned smoothly into a two-base timing attack with Dragoons and High Templar. It moved out as soon as it had three storms banked up. By this time Stardust or PurpleWave had usually contained the passive Astra and taken a third base, getting an economic advantage that had not yet translated into an army advantage. Astra's passivity meant that armies had not traded and were very large. This, with the broken Storm dodging, made its Psionic Storms devastating.
As with the proxy gate rush, this two-base timing simplified the game and allowed Astra to minimize interactions and decisionmaking, so it could just optimize its play for a narrow set of situations. As long as it executed the push well, it was in great shape to win. It did not need to skirmish. The game states it had to handle did not vary much. Simplifying the game against sophisticated bots was a pragmatic approach.
It nevertheless took most of the 50 hour contest for Astra to convert this plan into a winning evaluation run. By then it had learned to make Reavers and ferry them with Shuttles, but I suspect it was not doing this well enough to gain an advantage. Its winrates appeared to very gradually improve over this window, but I couldn't tell from the stream what it had changed.
After eventually passing an S-tier, Astra had a few hours to try to beat Pluto. This was plainly an impossible task for it, but I was surprised to see Astra suddenly stagnate with few changes in approach.
Astra's build order was pretty flawed, cutting workers needlessly for no advantage. It was a survivable error against Stardust forced to use its defensive "seal-clubbing" strategy, and against PurpleWave (which without learning data was constantly at risk of throwing an eval with goofy cheeses like its half-baked cannon rush). But against Pluto, Astra consistently hit the five minute mark with fewer workers, less army, no expansion, and often permanently trapped in its base. This was no less true in its 1000th game against Pluto as its first.
Astra's approach to StarSkirmish was in some ways unsexy: It simplified the game, won mainly in games where it could keep the game state simple, and against top opponents either steamrolled or got dissected in a short window at time. But winning StarCraft has always been more about bland fundamentals than fancy tricks, and I always love re-learning lessons about how the game actually is versus how we are tempted to perceive it.
as far as the on rails comment : there's no good way to know the provenance of an LLM output, but the LLM made bots generally look like very small, modular, sometimes functionalized versions of the human equivalents; the difference being that the LLM is calling the funtions often rather than a conditional loop in a main().
a factorio bot I worked on when Astra was first getting previewed was essentially a directory of very small argument-receiving functional and self-supporting programs that had very limited scope, and an agents.md file that directed the LLM to where such capabilities existed to preempt a directory search every compaction/context-swap.
so you'd have things like "python walk.py N 40" for 40 units of northern travel. a watchdog that waited for HP to drop or movement to become inhibited, etc.
My very first experience with LLM driven bot development in the GPT3 days created huge unmaintainmable monoliths that kind of worked due to TDD but were essentially spaghetti. LLMs have been trending away from those paradigms into small self-encapsulated programs. probably because those are less abusive to ctx count and produce better results since they lose/compact less.
There's also a pretty high profile StarCraft: BW bot right now called Pluto trained exclusively by self play RL (AlphaStar style).
Jay Scott put it well: Adding features lowers your winrate. Only iterating on them increases it.
Anthropic (Claude models): Claude Code
OpenAI (GPT models): Codex
Google (Gemini models): Antigravity CLI
Meta (Muse models): Muse Code
One thing we're learning is that harness is HUGE. The difference in performance for the same underlying model can change dramatically across:
1. harnesses
2. allowed tools (no web search/fetch - how does the model refer to documentation?)
3. thinking levels
Overall, this is a useful benchmarking approach. May take some newer papers and evaluate them this way.
I have shared examples that the tested AI models have written. It turns out that the AI models write code that does not resemble any of the bots in the training data.
Pluto is very good at assessing relative army strengths based on units within its vision and will always engage when it judges that its army strength has an advantage, and it uses superior micro and targeting to win battles efficiently. The key weakness of Pluto is that it has no memory of units that are not in its line of sight, so it can be easily exploited with fast skirmishing and harassing units, drops, and spreading out artillery units to deny it from seeing the full army strength all at once.
Its other big weakness is that once you get an army advantage you can force it into a turtling position where it will never try to attack you and you can safely colonize the entire map with expansion bases, giving yourself an insurmountable snowballing resource advantage.
From what I've seen: the Pro's wins mostly rely on exploits, not superior gameplay. Those who fight without exploits lose pretty bad, just in the last few hours: https://www.youtube.com/watch?v=xga636MBOns
Obviously chess bots can beat anyons but this involves a situation where you're making 1 move at a time in a highly constrained environment: exactly 64 squares and a set number of pieces and so in. You also have, relatively speaking, near infinite time to plan your next move and as a result AIs can use a combination of databases and simulation algorithms to derive the best next move.
Almost none of that applies to starcraft.
We can talk about "teaching" the bot to handle other situations but that word covers a gigantic realm of complexity, like how do we encode this information in a way that the ai can actually draw from to apply to its current situation? If people talk about "more chess positions than atoms in the universe" or whatever, this is so many orders of magnitude smaller than the number of valid starcraft positions that we need to invent new numeric notation to accurately describe it.
That being said, there is a substantial difference between humans playing at anywhere from 50-500 apm and a robot that can literally micro every single zergling on the map at the same time and do things like avoid ever taking the splash damage portion of a siege tank shot.
Around the time Starcraft 2 came out, I talked about it with a friend who was very enthusiastic about the subject. He said he thought the game was awesome, but it was really too bad that it imposed all this mindless tedium on you. (He was a programmer, of course!)
He said it would be really cool if you could just give higher level directives to your units. As though you were commanding them from far away. (The game, in his vision, would somehow account for speed of light, across vast distances.)
I think he really just wanted to play Civ or something :) But this post reminded me of that conversation.
- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
If that's the case then these models are failing extremely hard. 7% win rate against human written AIs that they should be copying from memory? A copy paste command should do 50%!
All of the above, to the extent that they can be measured from a simple to verify end condition (stats, win rates, etc). Adaptability, ability to understand and progress on a task, and so on.
The customer may not care what the internal contributions are to such a measure. Instead it represents the customers’ overall values in some way.
Worth digging into the protocol and see what went wrong.
It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.
Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.
For Brood War they're at https://www.basil-ladder.net/
I think with the abilities of AI improving, playing games would be a good benchmark as long as they're not trained on them.
Smells like fear
They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.
- BWAPI: https://github.com/bwapi/bwapi
- BASIL (24/7 ladder): https://www.basil-ladder.net/
- COG / AIIDE tournaments: https://davechurchill.ca/starcraft/
Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
that said, with a clock on this benchmark it's very possible that lower reasoning = more iteration = better code.
Making actually good decisions is the bottleneck.
Do you plan on comparing harnesses also?
Playing in pi, opencode Vs the original harnesses?
But right now the response time would be so slow that the bot would not be close to competitive except with the most low level players. Especially because its hard to click on things that are moving at the same time as well.
Sometime soon, I'd like to see an LLM be competitive using only computer vision, mouse movement, and locked into a human possible number of actions per minute.
I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)
You can train a specialized model for this, but a frontier model is ill-fitting.
TLDR they suck and many default to doing probe rushes.
- how do I train a model to play a 3D game?
- what is the process?
- what is the data needed?
- what type of neural network is required?
- think splinter cell if that helps
On a more serious note: all three races play quite differently so it makes sense to focus on just one. And protoss is mechanically the easiest and most straightforward race: fewer but stronger units, a single worker can be used to build multiple buildings at once, versatile basic static defence
Protoss vs Protoss only also helps downscope the problem given the time constraints