> For tasks that would take a human under four minutes—small bug fixes, boilerplate, simple implementations—AI can now do these with near-100% success. For tasks that would take a human around one hour, AI has a roughly 50% success rate. For tasks over four hours, it comes in below a 10% success rate
Opus 4.6 now does 12hr tasks with 50% success. The METR time horizon chart is insane… exponential progression.
Opus 4.6 had no issue correctly identifying and mitigating a hairy out-of-order state corruption issue involving a non-trivial sequence of runtime conditions from thrown errors and failed recoveries. This was simply from having access to the code repository and a brief description of the observed behavior that I provided. Naturally I verified it wasn't bullshitting me, and sure enough it was correct. Impressive really, given none of the specifics could have been in its training set, but I guess we're finding that nothing really is "new", just a remix of what's come before in various recombinations.
Anecdotal experience from my team of 15 engineers is we rarely get "perfect" but we do get enough to massive time savings across several common problem domains.
That’s what marvels me is how fast LLMs are progressing. And it still feels like early days (!).
For methodology, I would check out the METR website though, they’ve published their results.