Hacker News Reader

Comment by glouwbug | Hacker News Reader

We have this same point made every release. I wonder if models actually are getting better or if these companies are just secretly hamstringing the previous models before releasing a similarly performing model

View on HN

I think it takes a while to start finding the weak points in a model. It seems fantastic until you use it enough to find out that it hallucinates APIs, or can’t count letters, or uses too many em-dashes, etc. you don’t really know someone (or something) until you’ve lived with them (or it).

I think there's definitely some quantization or nerfing happening post-release -- I definitely noticed that with Gemini Flash 3.7, it was super fast and accurate at the start and a few weeks later seemed to run in circles and overthink even a "what are you doing" message, all while doing almost identical work. I find it weird that we're so far into the LLM era and the post-launch degradation is still something of a conspiracy theory.

Models are actually getting better.

While IME 4.7 and 4.8 were not appreciably better than 4.6, Opus 5 was noticeably better, and Opus 5.5 is significantly better than Opus 5.

Try using a model from a year ago.

What would that prove if we are testing whether they "are just secretly hamstringing the previous models" as GP claimed?

This would require a conspiracy since Claude models are hosted by Google, Anthropic and Bedrock. All these companies would have to be working together to secretly hamstring the previous models.

I'm not claiming that they do, but just to play the Devil's advocate:

Anthropic controls the full serving stack, so it can change precision, hardware, kernels and settings whenever it likes. Reducing the KV cache precision, for example, saves a lot of memory (and money). It may even be possible to quantise the model on-the-fly when loading it into memory. They don't necessarily have to upload a whole new model.

Even if they can't do the same on all platforms, we can't deterministically compare them anyway.

For the Bedrock and Vertex versions the customer is doing business with Google and Amazon directly. I doubt Anthropic's contract allows them to unilaterally "change precision, hardware, kernels and settings whenever it likes."

At any rate my point remains that this would require a conspiracy between 3 companies.

Welcome to the hedonic treadmill! Just keep walking!

Reply on news.ycombinator.com