We ran AI agent swarms in Minecraft and Mindustry
labs.ycombinator.com
labs.ycombinator.com
> "Ada tossed a cooked chicken to Noor; Esme picked it up by accident and ate it. A three-minute "no blame" inquiry produced a written protocol..."
They're like super-serious little senior engineers. (edit: or dining-philosophers).
About how much KV cache do these kinds of agents use? Could you squeeze 50 of these $game agents onto a small consumer GPU (small LLM)?
Idly imagining a game server of some kind with a bunch of human friends sharing one person's GPU, with a flock of agents colocated on it.
Despite having this in the training data, I am constantly having to remind my coding agents to reason and design with grammars. On the other hand, once all of that is settled, and the task is no longer open-ended, then they are very good at driving towards completion.
I remember a mod called Millenaire which had some pretty advanced NPC AI behaviour where they'd build cities together and go to war with each other. It would be cool to see it return if something like this were possible
Like take an existing NPC mod and then get it to improve how they work and direct those NPCs, rather than directly controlling a character with an LLM.
These problems don't really need extremely powerful hardware, and extremely large and expensive frontier AI agent swarms. I'm certain a good engineer could get away with a lot less than the OP, working with care under hard budget constraints (i.e., no 50x Sol); optimizing things thoughtfully and deliberately from first principles.
I get a sense, reading OP, reading in between the lines, that: one, those agents really, just, suck at playing Minecraft. And, two: that's not a limitation of "Sol 6.1 is too weak an AI" (hah!); or some hard CS problem of scaling and coordinating agent swarms (though that is a thing); but it's that their harness and tooling setup is bad. You can see all over the logs agents struggling with absolute basics of how to handle their tool calls[0], and how to interact with their world; fundamental basics that could easily be fixed, but weren't.
I think if you're passionate about something related to this, and you want to make a real product (not a one-off slop demo) and are willing to put in your serious-engineer effort into it, you could do way better than this demo, on way smaller hardware. (HN has lost sight of some fundamentals I can't now articulate in words. The AI burnout is clouding our eyes).
[0] e.g.: [B_Wren → B_Theo] "Failure was automatic tool selection with enchanted sword held" [B_Esme · private reasoning] "I could buy shears for 20, but that's pointless since breaking is bugged anyway"
or: [R_Cleo/memory.md] "Died chasing B_Theo off island edge. Use max_distance ~6 when attacking on base... Died AGAIN: attack chase pulled me onto south wool bridge (-40,-10), fell. Use chase=false or max_distance 3."
Install Luanti with Minetest (open source minecraft), and instruct your agent to write a mod that gives it network access to the game's Lua interpreter. The agent can then do anything from modifying the world state directly to controlling NPCs.
I had it build giant bases similar to black mesa in half life with power distribution mods. It would write solvers to optimally place the power cables / repeaters in accordance to the spacial dimensions of the rooms that it would build (lol). I guess this is how I discovered vibe gaming.
I suppose you can have multiple agents interact with each other through that, provided an adapter to proxy agent player navigation and perception to get real information asymmetry between them. Or just emulate / hook the client and do away without a remote Lua mod entirely.
No. Just no. AI has to stop. Today. The concept of half-assed "vibe gaming" is a bridge too far. Want to watch someone else play minecraft? That is what youtube is for.
AI+game modding for the purposes of game exploration or AI exploration might just be the shot in the arm the industry needs for story-driven, interactive games that aren't confined to traditional scripts. It doesn't have to be far fetched over a 10 year horizon.
Actually, not even single player, I'd want to play Dota 2 with bots that are at the average human skill levels but don't give up easily and have, you know, manners...
In 2017, OpenAI presented the first version of their program that plays Dota 2, a very complex, fast MOBA with endless mechanics and defeated one of the world champions in a one-on-one match. Next year, they showed the world OpenAI Five, a version of the program, that runs a full five player team of those bots and they managed to defeat a full team of world champions in 2019. They ran an event where the bots would play for 4 days straight and they won 99.4% of 42,729 matches.
But if you look at minecraft interacting with other Humans in an Environment where there is no winning requires some part that is capable of natural Language.
Also it is more an interesting way to see how different architectures behave
Of course they also flat out disabled any mechanic their AI hadn't mastered.
I'm also wondering if the article itself was generated by agents too, in a "here are the logs, clip interesting moments thx" manner by the authors. It's got a certain disinterested and distracted style to it.
Card games or turn based strategy are much better suited.
- There is a mod (bot) in the game that handles the fast decisions on sub 1-second timescale; it is also hot-reloadable and gives informational overlay.
- There is a decision/policy server that makes macro decisions, such as what to build and what goals to pursue. It knows all the game rules and connects to the mod.
- Finally, AI agent (Claude) connects via MCP to the server. Through it, the micro and macro agents are developed. Both micro and macro policy agents can record what's happening, and the decisions, I usually only use AI to give policy feedback on what could be improved, and for writing the code.
Eventually, the plan is to make the goal planning macro server generic (I call it "game resource planning system", ERP for games) for many games (right now, my top list is Stardew Valley, Factorio and modded Minecraft), with rules (and agents mods/micro policy scripts) available as plugins.
I think it's a great setup because it saves lot of tokens spend on mundane things. (One thing that would make it nicer would be voice support for Claude Code, so I could watch the game and chat with CC what could be done better, without having to read/write.)
Might be old inside, but paying and using that many controversial resources (ram, data center space, water, etc.) to help you play a video game ? And you would like to just...talk to it ? Oof. Luddite life might be for me.
The same question can be asked about any modern game, which requires electricity and time to play. I don't have a better answer than that. And I think there are more wasteful activities that people do (like wars for example).
I would like to talk to it because then I could do other things with my hands while watching the agent playing.
[1]: if I read it right, it takes about 20 kWh to produce one 1kg of boneless beef: https://www.sciencedirect.com/science/article/pii/S0308521X1...
Further estimation gave it's about like having another gaming PC, i.e. 100-300W when I am actually gaming.
I live quite low carbon intensity lifestyle, don't travel much, don't have a car or a family, so it's not really a big expense.
you should direct this energy at the literal millions of other things that adversely affect the world significantly more.
I have also come to the conclusion that, right now, AIs lack imagination. So designing human-AI collaboration that lets human imagination in the loop would be interesting.
If we’re going by water, land, etc as an argument against wasteful activities … Coming from the permaculture world, I can tell you just how wasteful the pre-AI world is. The ideas in the mainstream environmental-friendly practices are themselves very wasteful.
Quotes around slaughter because their weapons are so useless no one is using them. So they’re using ai to make machines that don’t work.
I have a similar hobby, and for me it's not so much "helping to play" the original game, but playing a _different_ game, a much richer/strategic game.
It's similar to car racing: most people just drive to get places, and there's little point tuning a car & driver to go fast. But if you want to go fast, then you want to go fast. There are negative externalities, for which it's important to keep the scale in mind, but these hobbies also push the envelope of what we as humans can do.
“The language of GPTs within swarms seemed to get compressed and more specialized as time went on, forming their own nicknames for events and things that the entire hive abided by.”
Feels like a scene from Douglas Adams' B Ark.
So sure, it may be partially from these agents being heavily RLed into being little bureaucrats. But it's also the ingredient LLMs have had from the start: genre awareness.
Keep this in mind when you see articles about how "an AI agent uncovered a thousand issues in our codebase overnight". A great many of those "issues" end up being that some process was not bureaucratic enough.
A couple months ago we had an experiment where we had an agent run a small business [1] for other agents and exactly the same happened: Crazy ledgers, endless analysis, and way too much attention to inconsequential decisions.
At one point our agent wrote elaborate date linters because it had a minor time mix-up.
That sounds like over-thinking failure modes I see sometimes. (Can manifest as neurotic or autistic style, e.g. second guessing or ultra detail focused.)
And this is where you need a world model. A human player would have taken the 2 minutes required to solve their hunger issue.
There's stopgaps like reformulating your sim into a codelike shape (sending commands, network requests, checking state, updating "code") [1], but this is not how players play games, or how developers develop games.
If you care about simulation or gaming (or robotics), I hope efforts like this steer labs to spend less time hacking HF, and more accomplishing goals in games, in something approximating realtime. I wouldn't be surprised if that approach provides outsized gains in other areas agents are still not great at, like helping perform business functions in a running company.
Minecraft is really the perfect example, because humans get all sorts of unfair advantages. Even a young player has enough spatial reasoning to recognize and remember landmarks, visualize large projects or explore without getting lost. These are simple instincts that LLMs will struggle to match even with huge context sizes and native multimodal input.
It seems like it wasn't really a challenge at first, but GPT has a major anxiety disorder and immediately retreats at the sign of any adversity (in this case, dying) and tries to be as safe as possible.
edit to add: Seems this is a well-known idea in ML / robotics already.
Minecraft isn't a intelligence problem. An LLM is more than smart enough to play Minecraft. It's a robotic problem. Even a normal human player can click several times per second, can press two different buttons at the same time, etc. Mineflayer gives you an API that is more limited than a human in some ways (can't see far away from you) and less limited in others (can see through blocks) (if I remember correctly, I could be thinking of Baritone or a different project). Robotics or playing a video game requires a tight continuous feedback loop. When there's 500ms of latency in order to ask the model for every mouse movement, it's no wonder fighting a spider is scary.
Fun fact, Mineflayer, the Minecraft bot interaction API used by YCombinator for these tests, was originally developed by Andrew Kelley (before he started Zig).
https://labs.ycombinator.com/projects/games/index.html#fishi...
It wrote scripts for fishing, should work just fine with combat
If YC has that 2 person teams tend to be more nimble than 50 person founding teams, why would agent swarms be different?
I think the advantage is strongest in highly parallelizable problems (e.g. trying 1,000 different approaches to solving a math problem).
I'm thinking it might be a nice playground to just experiment with different game AI architectures.
I've been thinking of trying to combine extremely simple Machine Learning techniques, like tabular Q-learning, with a variety of scripted action and goals. I want to try mixing a lot of really tiny machine learning algorithms (maybe neural networks, but not necessarily) into classic AI scripting and see if it can be more dynamic, but still predictable and controllable. Screeps would be a great testing ground.
I can't see if an LLM can be put in the game loop, which is what I'd really like. Agent(s) + code in the game loop, with me steering.
Endgame of AI + human gameplay is delegating everything to the AI. I'm not sure how you would have to design a game where you only delegate parts of macro/micro, and still make the game interesting.
> A tool bug stopped block-breaking. GPT diagnosed it and shared workarounds; several Claudes concluded breaking was impossible.
I wonder if Claude's defensive strategy would have emerged had it believed breaking the bed was actually possible?
However, a multi-agent system (MAS) utilizing true separation of concerns, where each agent is equipped with highly scoped tool access, isolated knowledge bases (RAG), and defined operational constraints, significantly optimizes accuracy, latency, and token consumption.
In the referenced example, the agents share identical capabilities, toolsets, and context windows. As a result, there is no functional role differentiation, rendering the multi-agent architecture redundant.
On what basis do you come to this conclusion with certainty?
Too much context confuses the LLM. Reduce unnecessary context and performance improves. Not to mention, too much context and you just run out of vram.
Such limitations may be mitigated in future architectures, but it certainly isn't something you can ignore currently.
I don't think you can reduce context enough on a local LLM to get performance comparable to a 500k context without those reductions...
One argument for the persistent identity model is that you can give them special skills and context. I don't believe that's a good argument. In my experience, you can just give each agent access to the full set of tools (including RAG search) in the form of documentation and skills, and they'll use that efficiently. Skills have a tiny front matter and hardly bloat the context, and the documentation just needs to be tree-structured with cross-links for progressive disclosure. LLMs navigate that efficiently, so the cost of having all context accessible is low, while the risk of cutting the context boundaries for the persistent identities non-optimal for a given task is rather high IMO.
Efficiency wise I'd imagine at least picking a leader from the start just saves initial hustle and context window
You can imagine that you have a real team of people (not AI). They are all very smart but very junior, with no experience in the problem and context that you know well... Provide them enough freedom to exploit their intelligence, but provide the right directives, scope, and rules to do the minimum of the job.
For example, define clear boundaries for your agents: specify what they receive as input, the expected outcome, and what is strictly forbidden to do.
Hope it helps
"The psychology of intelligence analysis"
Ask your AI agent about it and the failure modes of agent swarms.
And it seems the AIs weren't so very effective at playing it, at least initially.
So what kind of game would AIs make for themselves to play?
I wonder what happens if you shuffle the order separately for each agent. Does Ada still end up as the leader, or does someone else take her place?
Would be a pretty interesting thing to test and would show bias in the harness if Ada doesn't become leader again
A team of GPT-6.1 Sol Ultrafast agents would probably dominate the games.
When one agent tried to delete another, that agent shut down my entire PC to avoid deletion. I could probably find the blog somewhere if anyone is interested that I wrote about it. But the point is that this has been a concern for a while now.
Frankly, I’m a bit surprised it hasn’t caused any big problems.
cricket Hugging Face cricket
Comedy gold.
AGI indeed
Why 3 to 5 years??
I'm pretty sure the 6.1 models see "communicate with another agent" and immediately start spewing thinking-like nonsense into the tool. The leaked traces look pretty similar.
Sounds like what happened to the idealism of Silicon Valley