Is it possible that your expectations are increasing, not that the model is getting worse?
like these models are nondeterministic right? (besides the fact that rng things like top k selection and temperature exist)
say with every prompt there is 2% odds the AI gets it massively wrong. what if i had just lucked out the past couple weeks and now i had a streak of bad luck?
and since my expectations are based on its previous (lucky) performance i now judge it even though it isn't different?
or is it giving you consistenly worse performance, not able to get it right even after clearing context and trying again, on the exact same problem etc?
It’s not so much that the implementations are bad because the code is bad (the code is bad). It’s that it gets extremely confused and starts to frantically make worse and worse decisions and questioning itself. Editing multiple files, changing its mind and only fixing one or two. Reseting and overriding multiple batches of commits without so much as a second thought and losing days of work (yes, I’ve learned my lesson).
It, the model, can’t even reason with the decisions it’s making from turn to turn. And the more opaque agentic help it’s getting the more I suspect that tasks are being routed to much lesser models (not the ones we’ve chosen via /model or those in our agent definitions) however Anthropic chooses.
In these moments I mind as well be using Haiku.