Anyone with evals picking up a real signal? I've learned not to trust "vibes".
The only one I'm familiar with testing Opus 5.5 specifically over time is https://github.com/ninjahawk/livenerf and it's not showing anything.
The only one I'm familiar with testing Opus 5.5 specifically over time is https://github.com/ninjahawk/livenerf and it's not showing anything.
If you calculate Elo only before 11/3 and compare to only after 11/3, the Elo is 52 points lower after. Unfortunately we are still in beta so this isn’t conclusive (confidence interval of -160 to +40).
If we do this calculation on all our models, Opus 5.5 is first in apparent drop. (Again, not definitive, but supports the conclusion.
> livenerf makes everything else deterministic: frozen prompts
One way to test my theory would be to run the benchmark yourself. (This catches if they are targeting this one account running the benchmark, have to say kind of unlikely...)
The other would be to see if something changes when you include this in your prompt: (this is the system prompt used by this test)
```You are completing a task that is graded automatically and exactly. Work carefully. Follow the output format in the task precisely: graders do not read anything outside the requested format.```
To control for changes caused by the addition of this extra prompt itself, we could try to add some other piece of text that is semantically the same in meaning. (wont be perfect but maybe its close?)
To isolate impacts further we can do the same test with other non-nerfed models. (we ask, is Opus 5.5 the only model that is suddenly doing a lot better after i add this magic string in my system prompt, or will gpt5.6/quen 3.8 etc get better too)
Firstly, there is a lot of A/B testing. You would likely need at least 10 or 20 accounts to catch these random tests.
Secondly, the labs obviously run GPQA and MMLU like livenerf does and watch for their own regressions before doing A/B tests. These benchmarks are statistically the thing already being filtered out. You need tasks out of distribution. Ideally ones in distribution with the stuff they particularly frequently change (like safety or copywrite adjacent tasks).
These harnesses, and hidden system prompts, do get screwed with quite a lot in my experience. The models are actually remarkably consistent via API but there are tons of stealth system prompt and harness changes that lobotomize these models all the time.
Like despite being stochastic, try asking Luna to generate you a video game pitch via the API 100 times. It's almost always the same thing about lighthouses for some reason. That's actually why I made a CLI entropy generator project.
They also have this comment in getting started: "Pin the CLI. This is not optional: a Claude Code update changes the harness, and a changed harness looks exactly like a changed model. Turn off auto-updates and write down the version you're pinning"
Wild because as an end user I don't care what Anthropic changed to affect performance, I only care that Anthropic caused degraded performance.
So your harness behavior can change without updating it.
Fun.
The only thing I could maybe see is that any changes that are being made are happening on the subscription tiers on not the api tiers, but even that seems like quite a stretch.
Like anyone who is building agents with a good eval suite would know, models have been pretty consistent and even small harness changes can impact performance considerably.
If there was a silent degradation then the recent chatter about suspected nerfing is what you would expect to see. Of course that's not evidence, but I dont think we can say it's far more likely that its all subjective.
I mean the basic intuition doesn't seem unfouded. Maybe people can confirm/deny this contention for compute. But Anthropic is compute constrained. Inference takes compute. Model development takes compute. Isn't it plausible, and even likely, that there will be periods of contention? Would Opus 5.5 performance just silently get worse if they gave it less compute, as opposed to just being slower or something? Or do we have information that suggests inference compute is affected by model development?
> as of its latest test on Oct 16
- claude used first person pronouns minimally before ("I","my") and suddenly like an average of every 3rd response has them.
- claude introspection'chatter' has increased like 5x over last week ("I'm doing this now. I will run this. I will check that."
- tool calls are like 6x more than what they were. last week it was <prompt>, thinking, tokens, work, response. Now it's prompt, thinking, introspection "i'll do this", tool call, tokens, introspective response, tool call, work, tokens, tool call, ..., response.
Anthropic has confirmed that they make changes to the models, for example with the recent update to limit reverse engineering.
[1] https://www.rensoart.com/why-your-eye-is-ahead-of-your-hand-...
With Opus 5.5 I (subjectively) observed decreased performance across a variety of projects from greenfield to mature, across a variety of tasks, and a variety of days. Across a variety of projects is evidence against the claim that this is merely a model struggling as the project grows. A variety of tasks suggests it's not just "oh you're doing something you wouldn't have done before" or "your expectations are just higher now". Over multiple days suggests it's not just my subjective mood that is changing.
No one catching degradation in benchmarks doesn't really prove anything besides Goodhardt's law. It could mean that the performance truly hasn't changed, or it could prove that Anthropic runs the same benchmarks when verifying the performance. Just like I can have a fully passing unit test suite while my entire project crashes and burns, the benchmarks are only good to the degree they accurately represent real usage. Which in my experience they don't.
And finally, in this case with Opus 5.5 there are people finding concrete evidence of the degradation by analyzing their Claude Code session history.
So to my main point, I wish people would stop trying to gaslight us or defend Anthropic. No one outside Anthropic truly knows what's going on with these models. We have some direct statements from Anthropic. We have tons of reports online that seem to indicate tons of users (subjectively) experiencing worse performance all around the same time, and we have our personal experiences of whether the models perform the same or not. So rather than assuming everyone with a bad experience just has bad expectations or is some sloppy vibe coder, maybe we can just say "it's unclear what's going on." (Not saying your comment is pejorative though)
A large group of people reported that Claude Code's capabilities were dramatically reduced. And we heard then what we heard today; we need objectivity, we're being paranoid.
But there was indeed a bug, a bug that excluded thinking tokens from being included in requests after 1 hour idle. Reasonable for an old request one time, but the bug was that thinking tokens continued to be excluded even if you started a /new session, unconditionally.
And now we can't see thinking tokens and look out for this issue.
He has his "nerfbench" he calls it. Very useful and entertaining at least. :)