> Could you expand on your criticism?
Sure! The study focused on experienced devs working in complex codebases they already knew thoroughly. This was and is the worst case for using AI tooling from a cold start, _particularly_ AI tooling as it existed at the time.
There were also only 16 developers involved in the study.
Time has passed since the study and we've had an entirely new class of tool introduced (the agentic CLI a la Claude Code) as well as two subsequent generations of model improvement (Sonnet 3.7 to Sonnet 4 to Sonnet 4.5). Given that the results of the METR study were stated as an eternal, unqualified truth, the fact that tooling and models are much superior now compared to when the study was conducted is worth noting as well.