They mention that space bunny was an earlier checkpoints, so it may have improved (or may it could be benchmaxxed)
I find it interesting that so many models seem to be at almost exactly 74% specifically which makes me suspicious of the benchmark. I want to say 5.6-sol, Astra, deepseek flash 4.1, opus 5 (and 5.5?), Gemini 3.8-flash and a few others all ended up there?
There was talk of deepswe tests being flawed. I have yet to find a better gauge of useful work for my case though, so I still keep referencing it and sampling it. On the lookout for improved benchmarks.
"We found issues in at least 23 out of 113 tasks (over 20.3%), leading us to designate the benchmark as Flawed." - https://epoch.ai/benchmarks/deepswe/review
Models like Deepseek 4.1 flash, Muse 1.3, and MiniMax 3.1 flash are certainly capable and very cost effective, but I’m not sure anyone would say they’re better than Fable 5/5.1 in capability