Hacker News Reader

Comment by liuliu | Hacker News Reader

I haven't evaluated this one, but for what I have, it seem the pplx-decider-v1.2 (and its -flash variant) are pretty good. Anyone evaluated all these open-source decision models on their own tasks? (I evaluated on MonitoringBench).

View on HN

I didn't look specifically at open-source but I evaluated as many decision models as I could on openrouter against how good are they at detecting phishing, which isn't the best use of these models but I was interested to see how they compared. https://emdashcyber.com/phishnchips-decision-models/

I had my first use case recently, and did some very non-systematic benchmarking of models on OpenRouter.

The only one that came close to Jev itself was the one from respan (respan/span-02), which is also open source and cheaper too.

Have you checked mammoth-1/elephant-1 models? They are benchmarking quite well compared and are currently free on openrouter. I currently use mammoth-1 for a small internal app and it works flawlessly for what I am doing.

https://huggingface.co/baselabs/mammoth-1

No. I will, and I will report here.

Did some ad hoc/small test. Unfortunately not very impressive. At least no on German short text, which is my use case.

Ah damn and also interesting. I am using it for classifying ads in podcasts. I am listening to a lot of English podcasts, but I am in Germany so often German segments are cut in. Mammoth-1 has had zero issues classifying those segments, but I guess my usecase might just be a lot simpler than yours.

You might also be interested in this index for decision models: https://huggingface.co/spaces/multimodalart/jev-decision-ind...

Mine deals with sentiments of news headlines.

Thanks for the index. I will try to navigate it. For now I cannot even find this respan/span-02 in it :/

Reply on news.ycombinator.com