Yes, this generates probabilities over a set of given options instead of the whole token vocabulary.
I don't see what's so exciting about it, compared to regular LLMs which support structured output options in the API.
I keep evaluating Jev for my product because of the hype, but the reality is that for my tasks, Luna is more accurate, only a little more expensive, and the latency doesn’t matter. I’d rather spend the extra $50 / month or whatever than have to shoehorn in another API and provider, and also lose the ability to change reasoning level and get reasoning summaries for eval purposes.
But that’s just for my product.
Models like JEV fall in the golden middle. Somewhat cheap, somewhat fast, somewhat general.
Many people have replied saying that narrow classifiers (eg bert) are better, and they are better in many ways (essentially free and faster). But... the real world is messy, in my experience it's actually very hard to make a really good classifier that will work across a specific niche domain especially when there is very fuzzy input. And the most interesting real world scenarios are nuanced and overtime there will be edge cases discovered where it fails and that will require retraining and re-evaluation etc etc. It can becomes a full project on its own that eventually collapses into a game of whackamole (improved in some direction but regressed elsewhere). Things like JEV (or similar models) provide ease of mind, just off load the complexity to it and move on to the next challenge type of thing.
Doing better, I suspect, will be throwing a bigger model underneath and doing something diffusion like (we’ll allocate a whole block to compute rather than just the constrained space) that doesn’t compromise on latency.