Hacker News Reader

vengadanathanon Jeff – Jev-compatible 0.8B decision models, trained at home, ~30 ms

Well if there is compliance / data privacy need to it, it makes absolute sense. We operate in vocie ai industry and run in regulated environment. Jev etc doesnt support languages we want (indic languages) and for our usecase and it is not self hosted either, so running fine tuned LLM based classifier is more accurate and low latent (no external network hop as it runs in VPC). Idea is to re-use as much as possible but if compliance, accuracy becomes bottle neck it makes more sense to do it yourself.

vengadanathanon Jeff – Jev-compatible 0.8B decision models, trained at home, ~30 ms

No necessarily, we are getting similar result gemma 4 12b, predicting one output token for label. it is just prefill and with nvfp4 caliberated , we are getting 70-80 ms p90 for the classification workload we have.

not sure what made you think you cant go cheaper than jev. it is being done for a long time.