At 300 tokens per call, you'd get:
One million decisions on Jev cost about $12.60. One million decisions on Clef cost about $72.
Would probably make sense to self-host Clef, if you have the capability/resources. If not...
It's weird to think of these kinds of models as having "output tokens". Cross-encoder approaches like Laya add a [MASK] marker per option, but nothing is generated the way an autoregressive transformer generates. It's one bidirectional pass over your input, then a small head scores each option, so you wouldn't really pay for output as much as only input
Since it benches better than Jev, Jev is probably smaller and easier to host. They could also be losing a lot of money.
And while Jev has a lot of latency, that latency stays very flat with larger inputs. So it's probably related to their inference pipeline, not model size
For privacy perhaps but on pricing you're unlikely to come out ahead versus datacentres with scale and industrial power pricing
Turns out AWS is actually about 10 times more expensive than renting your own datacenter compute.
It’s like comparing pricing of a ribeye at the butcher and on restaurant menu. It’s apples and orange.
Certainly AWS is making good money too though. And yeah if you just need a VM then AWS isn’t the way