Artificial Analysis v5 and the benchmarker's dilemma
interconnects.ai
interconnects.ai
I get it, they can only benchmark so much. But I'd rather have a benchmark of how the model is actually used and offered to customers rather than the best case scenario for it.
Is this true? If yes it makes a lot of sense. Time and time again, when I try some newly released open weights model I am disappointed. Even some older Claude models are beating them on quality of answers, even though they score less on intelligence in benchmarks.
Most companies I've consulted for when it comes to LLM usage in various ways all have their own evaluations, not sure why others would avoid them.
This is to save time, for what it's worth. Doing that testing and evaluation will take you MUCH longer than doing so automatically, for obvious reasons.
Interesting. My experience is that Opus 5.5 is smarter than a model like Qwen3.8 Flash Next. But given the choice, I'd actually rather work with a model like Qwen3.8 Flash Next.
Even on medium reasoning, modern Qwen and DeepSeek models are ridiculously diligent. If I ask them a question about which library is best for a given task, they'll download 4 different candidates, read the source, and write test programs. Meanwhile Opus will look at a couple of web pages and give me glib bullshit answer. Opus is often like a brilliant coworker who can't be bothered to do their job. The modern Qwen and DeepSeek models are like someone who is merely smart, but who has an alarmingly good work ethic.
And I'm not just writing simple React CRUD GUIs. I'm doing stuff like modernizing ancient software forks, working on high throughput distributed systems, or finding bugs in other people's parser generators. And even the 120-240B open weight models are normally entirely adequate for these tasks.
One of the benchmarks where open models tend to perform badly is Terminal-Bench 4.0. I looked at some of the sample tasks on that benchmark, and some of they are really hard and specialized. Look, all I actually need out of a model is things like "tell me where this distributed algorithm might break", "get this ancient C code to build on modern Debian with modern dependencies", or "write a plan to migrate this database from MySQL to PostgreSQL and to test that the app still works." The modern open models can do all that, no problem.
It's worth checking different behaviours across the available models from time to time. What you end up preferring may be very dependent on what you do. For example while being diligent is nice in a lot of situations, sometimes I ask a personal agent a question and mimo spends 10min validating the answer in a few different ways. Sometimes I want that big research, sometimes I just want to know what that transformers toy is called.
There's also a huge difference is writing style. I'll take GPTs version of a commit summary over Claude and its copies any time.
I’ve been using Qwen3.8FN and DeepSeek. I would not trust them to evaluate 4 different libraries from scratch unless I gave them extremely specific benchmarks to run. They’re very good at consuming tokens for long stretches and then giving you what looks like thoroughly researched output, but their intelligence just isn’t at a level I trust.
But really, the difference you’re describing is easily shaped by prompting. If you want Opus to download and compare the libraries, just tell it to. It will. All of the models benefit from being clear in your intentions. If you try to find models that will read your mind where spending 20 extra seconds to type out your intentions would accomplish the goal, you’re ignoring the very important intelligence and efficiency differences between models.
> Look, all I actually need out of a model is things like "tell me where this distributed algorithm might break", "get this ancient C code to build on modern Debian with modern dependencies", or "write a plan to migrate this database from MySQL to PostgreSQL and to test that the app still works." The modern open models can do all that, no problem.
I actually read the code output for projects that matter. There is a huge difference between a model that can eventually accomplish a shallow goal like “fix this bug” and a model that does it cleanly. Even with Opus or Astra I will spot cleanup issues frequently.
When using models down to Qwen3.8-FN or DS-Flash size, the code can feel like the kind of spaghetti mess that you get from an overly caffeinated and confident junior developer who accomplished the goal via brute force and staying up all night. Yeah it compiles and does the thing you asked, but all of that spaghetti code compounds with each change until a couple months later your harness is spending half a day and countless tokens doing simple tasks because it has to navigate the mess of a codebase.
That problem is much less of an issue at the Opus level and I can trust Opus to clean up better when directed to do so. It’s those differences that compound.
But if I’m doing a throwaway task in a sandbox that just needs to work for a day? I’ll throw it to the open models and see what they can do. They are interesting to play with.
For things where I really care about code quality, I still prefer to write about half the code, including almost the architectural bits, by hand. The models are allowed to pick holes. They're surprisingly good at that.
I often use the smaller flash models. These may need a nudge once in a while, but not significantly more than what I need to do with the much more expensive Anthropic models.
The end product is often the same. Sometimes a model surprises me with a specially clever solution but this is random and can happen with any model, open or closed.
My experience with Chinese open models is that they couldn't cope at all with my production work; both GLM 5.3 and Deepseek v4 ( or was it 4.1? Couldn't remember too well) went into infinite loop and wasted my tokens until my OpenRouter wallet reached 0; good thing I didn't enable the auto topup. US models, by contrast, breezed past them.
Even for simpler tasks, Chinese models took long time to complete, and I needed to supervise closely. The price for Chinese models, in the end, didn't come cheap, mainly because too much time was wasted on thinking, and this was regardless of whether they finished the tasks or not.
The article is correct that the major labs are not benchmaxxing, but I think it overstates their focus being on something else.
They don’t play the benchmark manipulation game because they’re comfortably in the lead and they have their own teams producing internal benchmarks and targets that are representative of what their users expect.
Your experience with the open weights models is not missing anything. Nearly all of them are disappointing if you set your expectations according to where they fall in the benchmark charts. They can be useful for some tasks once you’ve developed an intuition for their capabilities, but you have to start learning their relative cost per task and their limits, which takes a lot of observation and honestly might not be worth it.
One of my unpopular open weight model hot takes is that, right now, OpenAI’s Luna model is often cheaper per task than even DeepSeek and others (paying API prices, not based on the fluctuating subscription prices).
The real advantage of open models has always been that you can run them on your own hardware or a rented server and have more control over where your data goes. Some of the hype about them being 3 months behind frontier models or people making claims that 27B local models have replaced their Claude subscription become extremely questionable if you actually use both open weight and frontier models frequently.
Whatever they deem as the next big thing ends up being emphasized in the public benchmarks.
Google and Facebook are doing this, but these two were also considered to be the worst in class among the US labs until very recently.
A and O can't do this. And as we recently saw with fable, Anthropic can seriously benchmax their releases.
If we had access to neutral raw benchkmark data in a useful way to compare apples to apples, then we could each define our own 'index' to get the slice of perspective we find useful.
Trying to define or compare 'intelligence' through a single index will always be off.
its too much work for 98% AI consumers. In reality, AA is best what we have as aggregated score..
maybe (this could be discussed), but the problem there is nothing better..
So you really need to develop it yourself. If you buy such benchmark from someone, and the benchmark is any good, you may bet the labs will be buying too
For the time being, LiveBench seems to provide a better calibrated signal.
They haven't received much of investment.
Example could be, we'll update X criteria when top-N models are within 2 points of each other.
The numbers are fuzzy and hard to verify, especially if benchmarks are private, putting authenticators in a powerful position to determine market value of AI in countless ways; a recipe for corruption. I'm not saying it's happening, I just realized the social ingredients and temptation are there.
It’s the crow of current LLMs: tiny brain, outsized performance.
Not sure if the scores are better, but the website is nicer to use.
AA has such terrible UX, especially on mobile, trying to select a subset of models you care about is an exercise in frustration.