Hacker News Reader

Comment by sebmellen | Hacker News Reader

> it runs up to 1.7× faster than comparable MoEs on-device, with 184 tokens/s on an Apple M5 Max and up to 21K output tokens/s on a single NVIDIA B300 GPU

Wow, those are seriously impressive numbers for how intelligent this model seems to be!

One thing that this article doesn't clarify, and that I guess I really should look into, is what exactly AutomationBench tests for as compared to other benchmarks. The standout performance on AutomationBench is interesting to see, but I don't know what it really means because the model seems to score relatively close to other local models on every other benchmark.

View on HN

I'd guess it's mostly just an artifact of the strength of their Agentic RL environments for those specific domains. If they had the same kind of access to massive amounts of institutional software knowledge, compute infrastructure, and hundreds of millions of users dropping them endless highly technical agent traces, there may have been similar disparities on the agentic benchmarks in other fields.

It's possible that they may have generally capped out for some reason... but, I'd rather be bullish on a new American AI lab making it to first place among western open-weight models that can run on a common consumer device. (Closest in size that's better across that range of benchmarks is Inkling-Small I think, at 276B params, unless I'm forgetting someone.)

Inkling last I used it was pretty terrible. I've never paid for a model that got stuck in a token puke loop and couldn't do tool calls until that day. Even the dated qwen models from almost a year ago 1/10th the size are a better user experience than the inkling model I piloted. Hoping they improve, sincerely, but not a good experience at all.

What does it matter what country an open weight model comes from? Please don't say you tried asking about tienamen square to a model 2 years ago and this was so scary you couldn't use it like 85% of people with this take have said...

--It scores only a 6 on the "Artificial Analysis Intelligence Index". That makes it basically useless.--

Edit: I was wrong, I looked at the wrong model

That's LFM2-24B-A2B, a non-reasoning model released in November 2025 (https://artificialanalysis.ai/models/lfm2-24b-a2b). This model, LFM2.5-24B-A2B, hasn't been ranked by AA.

A sorry my bad. Thanks for pointing that out. Then I am pretty curious what it will be.

Reply on news.ycombinator.com