Hacker News Reader

Comment by lostmsu | Hacker News Reader

Is Jev any faster, cheaper, or more precise than medium LLMs like Luna 6?

Yes. Not as fast/cheap as Typesafe claims, but lots of benchmarks suggest around 3-4 times cheaper, and 7-14 times faster.

- https://www.ml6.eu/en/blog/jev-vs-gpt-6-luna-vs-bert-text-cl... - https://tessl.io/blog/jev-is-136x-faster-and-27x-cheaper-tha... - https://x.com/fazxes/status/2100300097695232164 (this last one is Luna 5.6 but that isn't too different from 6 besides accuracy and cost)

If you can do it offline, batching solves this. We used a test dataset from CFPB and at n=20, it was 1.6x faster and 1.2x more expensive with no statistically meaningful accuracy dropoff. Did not tune for batch size, but it's possible that we could get to 30 and see better perf.

That's not one if, it's two big ifs. If you can to it offline. If you can batch. I wouldn't say that's all that common in the use cases Jev sells itself for. You often want the answer now, not "averaged across 20 items in a batch that took 20 times longer".

It's fair to note it, but I wouldn't say "batching solves this" so much as "batching can solve time insensitive use cases".

No shit its cheaper and faster, thats the case with every small model.

It's a trade off between wanting something dumber but fast and cheap, or something slower and more expensive that is vastly more capable and smarter.

But the thing is that across a lot of classification benchmarks, Jev is actually faster, cheaper, and smarter than Luna (at least without reasoning). For example, look at this one from another user in this thread [1]

[1] https://hard-decisions.anth.us/models/

The reason the comparison is being made is because Jev and Luna 6 have similar levels of accuracy...

Luna 6 is 10c per million input tokens and charges 5x that for output. Not sure if it’s still true but it used to be the case that structured outputs took time to process and cache which is relevant if the structure changes. It doesn’t give a percentage you can use for thresholds, and I’d want to know if Jen treats the questions as independent (they aren’t with luna, order of questions will change the result).

Jev is 4.2c/m tokens in and free out.

not really

I don't think Luna is fast enough by any means

Reply on news.ycombinator.com