Qwen4-Max
qwen.ai
qwen.ai
For comparison, here's one I got from Qwen3.8 Max (at default reasoning effort): https://simonwillison.net/2026/Jul/20/afraid-of-chinese-mode...
This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.
Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.
Here are those 3.8 Max 0902 pelicans: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... - they're mostly not as good as 4, but I prefer the 3.8 xhigh one.
You are not the first one to think of this. And Simon has thought of it as well, and commented on it several times.
It clearly hasn't; otherwise the quality of the pelicans and bikes would be static over time. Instead the quality has been improving along with the models.
By comparison opus 5.5 (max) is $10.01.
If they are not testing Chinese models with less than max thinking, then we should only compare them to other models also at max thinking.
They compare different reasoning levels.
https://artificialanalysis.ai/models/deepseek-v4-1-flash-101...
Scroll down to the cost per task pareto front chart which includes more difficult levels by default: https://artificialanalysis.ai/models/qwen4-max#intelligence-...
Qwen4-Max is not in a good spot on that chart. Astra xhigh and Fable 5.2 medium had a similar cost per task with higher intelligence ratings.
If it's actually there, what is the cost at medium thinking? It would show Qwen 4 Max (medium) if it's there.
Everywhere that I can see on the page it just shows ”Qwen 4 Max" while for 6.1 Sol, Opus 5.5 there's always a thinking level in brackets after: (low), (medium), (max) etc.
It's possible that they will run Qwen 4 Max at other reasoning levels in the future, or it's possible that they never will. Regardless, it's unlikely to be better at any reasoning level on a price/performance basis than the GPT 6.1 Sol/GPT 6 Astra pareto frontier. It's close enough to Claude Opus 5.5 line at lower reasoning levels that it could potentially cross that if it doesn't loose too much performance at lower reasoning levels. But it's far enough from the global pareto frontier that it's unlikely that it would cross that at any resoning level.
You can see for example that the cost per task for 6.1 sol is about 4x higher at max compared to medium thinking level.
Which effort levels are cost-equivalent is debatable and depends on whether you are paying API prices or on a subsidised plan or are able to run the models yourself on your own hardware. But I think AA uses public API pricing for everyone as a sensible baseline.
It does not (universally) hold for long running workloads -- which is what people are increasingly doing.
Leave a Sonnet on medium, it will complete the task, spend more time and possibly significantly more token money, vs. Opus 5.5 on xhigh.
I always use Max effort in both Opus and Fable. It's actually faster overall because tasks get done faster with fewer "corrections" from my end.
On GPT models, I'd sometimes dabble in xhigh/high but that's because OpenAI usage quota are too limited.
This is wrong.
To make a vacuum analogy, one brand's Low setting could be another's Medium setting.
To compare you either give them the same resources ( Watts or $ ) and see how much they can do, or you give them a task and then check how many resources they required to finish it.
The label on the knob is irrelevant, except that either model might have a setting higher than the other which is sometimes useful if the resources aren't a constraint or lower when they are.
Nah, there are interesting conclusions to be had in comparing various model/effort level combinations, esp. when it comes to cost and time per task.
I have my own private benchmark suite; it involves algorithmic problems, the need for good visual understanding, and the ability to run long workloads unattended (as the agents need to run a few iterations, self-evaluate the results, and hand over the assignments when they feel they have succeeded). Chinese models do poorly on such problems, and running them at anything less than max efforts is pointless; they can't even hand over the assignments graded against a mechanical set of tests they themselves can execute. Hand that same task over to, say, Sonnet medium and you have a weird outcome: excellent result, but way more time and tokens burned than, say, Opus 5.5 or Sol 6.1 running at high. I do those benchmarks just to be able to spot such patterns and then better choose the models for my real work - but i wouldn't have caught it if I wasnt running various model/effort combos.
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
They have altered their benchmark multiple times in the last 4 weeks (v5 in https://artificialanalysis.ai/articles/artificial-analysis-i...) and the last 5.0.1 change https://artificialanalysis.ai/methodology/intelligence-bench... pushed up Anthropic's models a bit, you can see that in v5.0.0 at the first link, Opus 5.5 has a lower score than in it gets in v5.0.1, and v5.0.1 was released just before Fable dropped.
Some part of me believes that the benchmark mix was put together to favor Anthropic's models.
But admitting their model is weaker is a no go. So conspiracy theory here we go
It's mainly just a composite of existing benchmarks. It would be very easy to benchmax if, say, AA shared with a preferred lab which benchmarks it plans to upweight/downweight in the next version. Just speaking hypothetically, of course.
Also, a generic benchmark might not translate well into what everyone needs. I keep building my own test cases that test models for my own work specifics. Only that benchmark is useful to me. I don't need it to win a PhD math quiz.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
Maybe I'm mixing up model variants, but Qwen 3.8 27B that I run locally totally does handle images. Surely the bigger one would as well?
Instructing it to ask any and all questions as necessary sends it into an existential crisis, because the question tool (oh-my-pi harness) suggested limiting the number of questions, and it couldn't figure out how to do both.
Looking forward to the "Flash" and 27B versions.
This forces ant/oai to lower their prices but how much that is subsidized nobody knows. It might very well just be they are in deep trouble, they see their clients leaving, they have to lower their prices.
But let's not forget openai is riding on the edge of bankrupcy, their last investment round apparently gave them few months of runway, they are leaking money. I don't think they know how to turn this around with all the open models being competitive.
I would just enjoy cheaper model costs while they are like this.
I feel open-weight was able to keep up the past year using efficiency improvements (hybrid attention, improved MTP drafters, improved optimizers) to offset having less compute. But OpenAI/Anthropic started cutting token costs just a few months after these, which to me suggests they copied these improvements. Indeed it would be hard to work in LLM training and not know about DSpark, Muon etc.
So what do you do if closed source uses your learnings, gives no knowledge back, and has 10x more compute and funding on top of it? The situation feels quite dim right now, esp. with the Nvidia thing.
If we extrapolate over the next several years, prices for VRAM/RAM will start to come down as manufacturing capacity comes online, models will continue to become more token efficient, and intelligence will improve at every quant size. As this happens, more and more people will be running LLMs locally on hardware they already own, and there's going to be a point where local models can do everything you need to do unless you're literally trying to do breakthrough mathematics research.
> and there's going to be a point where local models can do everything you need to do unless you're literally trying to do breakthrough mathematics research
Maybe. But competitive markets pay exponentially more for incrementally superior services. Professional athletes are a great example of this. If you're building a product with competitors and those competitors have frontier models and you stick to local then you're probably going to lose both today and in the near-ish future.
If you're bitter lesson pilled then the alpha is entirely in having a better model, more data, and more compute. Your vision is that human ingenuity will provide more alpha. Oof. I wish. But my tastebuds are very very bitter.
i highly doubt this premise. the success of a product has very little to do with the amount of the intelligence used in producing it. you could throw millions of dollars using AI to develop a movie and it would still flop in comparison to an indie-made film.
Once I have a model locally, I know it isn't going to change on me. Compare that to using an API from Anthropic or OpenAI (and yes, the only thing you can say is you're using an API, not one model), they can discontinue a model you're relying on (as every model could have peculiarities you've designed around) or modulate their speed/quality day-to-day to save on compute costs.
And trying to claim that buying your own hardware doesn't pay off compared to API or subscription costs doesn't hold much water either, at least not for me. For starters, I already had the GPUs and machines I started running models with, whether it was for gaming or for other reasons. And in the case of the new GPUs I bought recently (2x AMD Radeon Pro R9700; no you do not need CUDA and you haven't for a long time now), I can hold a reasonable expectation that they will hold at least some of their value years from now, compared to paying for API/subscription usage for years in which case you're left with nothing.
So, if you have a local model that is "good enough" (and they already exist for many use cases), it's just kind of the obvious choice to make, IMO. And in just one person doing it as a hobby while I'm still looking for a job 16+ months after a layoff. A small business and certainly a large enterprise has even less reason to open themselves up to all the downsides that come with using Anthropic or OpenAI.
.. if exponential, they’ll never catch up but if logarithmic, closed models will plateau and so catching up is inevitable.
My money is on a plateau
Following that logic the gap widens a little bit here but then will steady again. Before this there has been a general catch-up trend too which probably also persists leaving us in a slow closing of the gap as the net effect?
This doesn't take other factors into account. Compute capability, infrastructure capacity (the chips can't work without electricity!) etc.
Those factors translate into a ceiling for this hypothetical exponential speedup.
So, with new models coming out on a weekly basis, I think trying to figure out a future trajectory is just going to be a wild guess. For all we know, Anthropic and OpenAI models may also be plateauing in the near future; or there may be a breakthrough and they may improve beyond our wildest expectations. I honestly don't know.
That said, I suspect available compute and model sizes that major Western vs. Chinese labs can handle will still keep the gap open at least in the short term.
On the other hand, I am also not sure how much the gap will matter past a certain point. I am specifically wondering at what point people will become interested in smaller models fine-tuned for specific application areas (e.g. cheminformatics) rather than massive generalist models.
Their open weights models are the the smaller ones, which aren't released yet.
There was also a bug in which the reasoning tokens were “encrypted” for opus but you could ask Haiku to regurgitate them and it would.
Anthropic have been doing anything and everything to prevent distillation “usage” and it’s caused me at least once to cancel my subscription because it’s hard to trust a company that secretly alters its responses based on something I can’t see and never get feedback on.
Edit: In addition to reshuffling the weighting of individual benchmarks, they also do things like test Anthropic and OpenAI in their native harnesses, while not doing the same for Qwen/Deepseek/etc.
Yet every time I compare the two in something like Pi, Sol ends up being 5-10x the price for a simple query, something like 7-10c vs. less than 0.5c for DeepSeek. What's the deal?
Models often get great scores despite being a lot worse than the frontier in practice.
Edit: I do agree their harness methodology is pretty bad, but I think that's more directly attributable to the fact that harnesses appeared after their initial methodological design, and they are too lazy or resource constrained to improve their methods. Still dishonest, but a simpler motive.
AA has always been meaningless, but people keep referring to it anyway just because the synthetic view it provides is too convenient (even if meaningless).
Ah yes, the reverse-benchmaxers of OpenAI models.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate number.
Maybe somebody should archive the scores and see if they in the future ever backtrack particular models to favour others (like last week)
Hoping it’s a great local model though!
I wish there were more ~60b dense options like ripple too!
Do you run Flash Next or 27b on yours?
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
American history is open-weight. American trade is closed-weight.
Anecdotal though, N = 1.
At least it doesn't call the cops on you if you try, unlike Anthropic.
It also applies to languages, I use Qwen 27b a lot and it's a lot smarter in english than in french.
Why do you ask? About half the time a china model gets released someone askes this. Almost as sure as a pelican.
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
I would assume it will not be on Bedrock for some time, AWS has not kept up with the latest non-American models.
Which is why the Cohere FUD was so efficient. That's a bullshit claim, but a bullshit claim that cannot be proven wrong can still do massive damage.