The FAQ of the actual study contains a lot of useful info that addresses some of the comments here, for anyone curious:

https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...