Hacker News Reader

The other criticisms are valid but this one is too much:

> No comparison to astra/6.1 sol/fable 5.1/opus 5.5

I mean, should a 27B model beat Fable 5 but not Fable 5.1 that would still be an incredible achievement.

The problem is that it's a very simple, older benchmark, and qwen was trained after fable 5. You need to test against similarly modern models so you aren't putting the models on uneven playing fields in terms of contamination. And for the stuff qwen is trained to be good at (agent stuff) opus 5.5 is light-years ahead of fable just because it's a newer model amd they RLed on it.

Though point taken that in a case where they didn't even bother making sure they were using the same qwen quants for every run or even test fable in their harness probably that is the least of the issues here.

Reply on news.ycombinator.com