Testing Jev and other decision models on search-related tasks

One of the authors here. I work at Parallel. We got Jev-pilled a few weeks ago and have been trialing decision models nonstop across all sorts of use cases, so we wanted to see how the Jev-like models compare. The tasks are a mix: some where we previously used fine-tuned models, some where we used general LLMs, and some we hadn't pursued at all before Jev. What I think makes our analysis a little different is that we tested on human-labeled data from our production pipeline rather than synthetic benchmarks, which can be made to show almost whatever you want (I suspect that's how so many OSS models were ranking above Jev in early evals).

What we found:

The ergonomics of serverless decision models were really hard to beat. Cost, latency and accuracy were all near the top for Jev. Latency especially was very consistent, which matters a lot to us; historically that's where we'd have reached for a fine-tuned classifier.

Perplexity's decision model was a close second. It sometimes outperformed Jev, but its latency was more variable. That said, I believe it's also half the price now.

Self-hosting open-weight decision models wasn't a big unlock. In principle, at peak utilization we'd get into the ballpark of Jev's API performance. That said there is a world of optimizations that we just didn't have the time to pursue.

Iterating on prompts got meaningful improvements out of almost every model. That's much nicer UX than having to retrain a classifier.

Caveats: there's already a new Jev out, plus a few other models, and we can't keep up. You should ultimately try these on your own tasks.