Hacker News Reader

Comment by mppm | Hacker News Reader

It's been suggested elsewhere that the AA-Omniscience benchmark [1] is more representative than the rest. It puts K3.1 about 6 months behind the frontier and the new version of DS 4.1-flash about 12 months. That roughly lines up with my gut feeling as well, at least when it comes to the trickier problems. Still rather impressive, but not quite frontier.

1. https://artificialanalysis.ai/evaluations/omniscience

View on HN

Huh, this is an interesting hypothesis! It would make sense, to a degree. Part of "big model smell" to me is the model being able to pull a huge range of interesting algorithms, architectures, mathematics, and everything else that would come with greater world knowledge and having a much more sort of innate, like, full, well-rounded understanding of it instead of surface-level ICL pattern matching on web results.

I skimmed through the publicly available questions. The first programming-related one I saw has the answer in the question, not a great start :D [1]

  Q: In C23, attributes are allowed in an unbraced (forward) union declaration only in the form “union [[...]] Tag;”. What single character must immediately follow the identifier in that declaration for the attributes to be permitted?
  A: ;

[1] https://huggingface.co/datasets/ArtificialAnalysis/AA-Omnisc...

Edit: The other programming ones look fine, bad luck I saw this one first I guess.

+1 for the omniscience benchmark!

Although even this one needs to be taken with a grain of salt. Fable/Opus still act very differently to Astra and Argon imo. But maybe it's just because I am used to these. I wouldn't clearly put them above each other though, although I still prefer anthropics models, but that could be simply out of comfort.

That definitely matches my experience, I have found Astra to be more capable than many other benchmarks show, have been very impressed with Argon for text, and generally like Sol. I find Haiku and Sonnet to be not worth my time as compared to the speed and cheapness of something like DeepSeek.

The only exception on the chart is DeepSeek - it scores exceptionally low there, which doesn't exactly track my experience. It usually does what you tell it to, and it's not exceptionally intelligent, but I do find it quite good at what it does (the bulk of code, scripting, etc). I know a lot of folks like it too ... curious if I'm just not exploring the simpler models enough?

Yeah that tracks completely with my experience as well.

Reply on news.ycombinator.com