Hacker News Reader

Comment by oh_no | Hacker News Reader

> this full release warrants a major reassessment.

I really don't think it does, the benchmark numbers are dreadful and the cost is higher than Sol. If you have to use an EU model knock yourself out but there are a ton of models that are a lot smarter for a lot less money.

View on HN

You mean they didn't benchmaxx it? That's an extra reason to give it a try.

“You know how Einstein got bad grades? Well MINE are even WORSE!” - Calvin, to Hobbes

Except he never did get bad grades.

So if scores are good, they must be benchmaxxed, and if they are bad, they must not be benchmaxxed. So basically, the narrative is whatever you want it, no matter what the data says.

No, if a model scores good on benchmarks it could be that they have benchmaxxed, if a modern model scores bad on benchmarks it could be a really bad model but it almost certainly isn't benchmaxxed. It's relatively easy to make a model do well on particular benchmarks, companies that don't care about that stand a better chance at making a model that does well in general because tweaking a model for benchmarks usually is at the expense of something else.

I run my own benchmarks (this model is - unfortunately too large so that won't be happening here) on data that no outsider has ever seen and this gives sometimes results that are not in line with the order that you would expect if you took the various benchmark sites into account. It is of course entirely possible that our benchmark is not representative but it seems to work quite well as a predictor for which models have value for us and which do not. Recently we looked a model that has been pushed with a significant amount of hype and for the life of me we could not get it to perform even close to the published benchmarks, whereas the models we used for controls were definitely within the margin of error. It's not always equally cut-and-dry.

I don't quite believe the benchmarks, especially supplied by the company.

Qwen3.8-Flash-Next consistently benchmarks better than both GLM5.3-Flash and the previous DeepSeek v4.1 Flash 0731. The DS 0731 was so atrocious it was getting beaten by small Nemotron in my coding tests (subset of tasks from open source coding benchmarks).

Now, new DeepSeek V4.1 Flash 1017 really gets the numbers they promised for the first one, but it's too big for me to run the 4 bit quant locally at interactive speeds.

So I kept Qwen3.8-Flash-Next as my go to.

Reply on news.ycombinator.com