DishwasherBench – Can Frontier Models Stack a Dishwasher?
github.com
github.com
I can't. So that would surely be proof of AGI.
I think optimal configuration is very much a NP complete problem.
It’s an interesting problem to pack the dishwasher with as many objects as possible and maintain the highest level of cleanliness. Sometimes, there’s an small thing leftover so then repack in a different configuration.
I did, once ;)
At home I'm the architect, my wife's the raccoon. I don't actually believe that you can stack the dishwasher wrong as such, but you can make it hard for yourself to unload, if you don't sort the items correctly.
There are definitely more and less correct ways, but personally i think it's "wrong" to load it in a way that leaves the arms blocked or (not) tilt pieces that get excessive water pooling on top
You most definitely can, they're not magic! The water must hit the right spots and obviously stacking three plates without any space between leads to stuff not getting properly cleaned.
If you try to teach a kid to put the dishwasher you'll learn a ton of wrong ways of doing it! :)
Dishwasher physics matter. But I would never criticize a person in my family for loading the dishwasher the "wrong" way; I just sneak in and reorganize when nobody's looking.
This is the way. I am just happy when someone else, anyone else, actually puts dishes in the dishwasher.
I learned a long time ago that the ability to recognize and exploit a structured system is the most emotionally satisfying thing in the world.
I read our dishwashers manual and was surprised at the significant amount of detail about how to position silverware for optimal cleaning (e.g. where to place small spoons, where to place forks, where to place large spoons, etc.)
Stacked loads are run against a dishwasher simulation calibrated to my own Bosch dishwasher
Playable by both humans and AI
Currently a WIP, simulations need refining before running a full benchmark, but keen to hear some feedback!
I suggest you have it scrub all of this from the readme; you’ll likely need to give it a few examples. It will improve readability a lot, and be less fatiguing to read. Plus if you scrub it all you won’t reprompt that attitude every time an agent reads the readme.
Btw I got Claude to give me the name “complaint smuggling” by giving it ten examples and asking it for a name. it’s an error pattern that’s so common I wanted to know what the model thought of it as - makes communicating about it really simple.
Thats the most common problem in my dishwasher. It kindof looks like it would simulate that but they dont spell it out.
Come to think of it, if the players dont have an option to "spin the spinner by hand to make sure its not catching on anything" they're going to be in for a rough ride.
> A spray arm held still by an item: the plates above it keep their verdicts
I wonder if anyone has crafted a LLM benchmarks list website/github.
my feeling is that stacking a dishwasher involves enough image processing and computer-use (or more realistically robotics) that it's a frontier model job rather than a 'write a program to do it' job
but keen to hear if you disagree
Imagine creating real simulated env for every physical task and using RL to train robots. That will be a game changer.
I calibrated it against my actual dishwasher by doing some test runs with baked egg
Sooo, the score is maybe mostly just how much they fit?
The sim needs work, for sure