More thinking isn't (always) better

Hi all! I’m building https://fairbake.dev (the business around Bakeoff) and when the question came up again in one of my calls of what effort level to set on model X for new {feature, bug fix, agent, categorizer, …} type tasks, I figured I’d try to answer it quantitatively using the eval framework and challenges I’d built alongside my clients.

When I dug into it, I realized that the models shipped at different default effort levels. That seems confusing to users - I wonder why the labs do this. I was surprised to see that increasing the thinking level often didn’t improve correctness and almost always increased (non-test) diff size. I was less surprised that Fable at high and Opus at max did poorly.

Ask me any questions you have; happy to share specific differences I noticed between solutions. What effort level are you running at and why?