There is no single opinion, and clearly no single Chinese approach.
Also Chinese labs are in particular very careful about anything which be used to create pornographic content, which is highly illegal in the PRC.
23,652 karma · joined December 5, 2014
Member of Technical Staff, AI @ Verda https://verda.com/
some src @ https://github.com/rdaum/
There is no single opinion, and clearly no single Chinese approach.
Also Chinese labs are in particular very careful about anything which be used to create pornographic content, which is highly illegal in the PRC.
The implosion of hype after the .com crash was actually kind of a ... relief.
Whereas the other guy went through the unglorious but formerly respectable path of publishing software and papers for other professionals to look at. A year ago.
We're in a bad place where the latter looks less reliable than the former.
(EDIT: I'm not saying the research here is in fact the same as what "Jev" is doing; and Jev is in fact more "product shaped." But I think it's important to temper the hype and back up and focus on the fact that this whole industry is built on research by both academics and enthusiasts ... first ... and gold rushes can often bulldoze over those people who are focused primarily on making-doing-researching instead of fundraising-hyping-promoting. That's not good.)
But only works for Gemma4, or other attention-only models. (i.e. not the Qwen3.8 models I had working with the other way)
I'm getting 63ms per answer for 1 question, 79 for 3, 93.7ms for 5. So scales nicely, too. ~55ms fixed cost + ~7-8ms per additional question.
Much better. Thanks. I'll commit and share work after testing it some more.
EDIT: how can I credit you in the commit body?
What I described was -- we have a bunch of orders to the kitchen, all of which have the same "base" meal but different topics.
> Cook the "base" meal once, divide servings onto multiple plates, then add different toppings to each plate.
vs what you suggest:
> Put the base meal and every topping through the kitchen together, but use some kind of dividers so the toppings never mix up together.
Except.. ok, that analogy is confusing lol.
Except the problem is not restricted to the actual ISA or its HDL implementation, etc.
It's even just getting space / time in a fab at that advanced of a process node.
Congrats.
Instead I use a fixed set of up to only 64 single-token labels. At model load, I gather only those 64 rows from the vocabulary head into a small matrix. Each question maps its permitted answers onto some of those labels.
It is a probability distribution conditional on the allowed labels. Calibration is a separate problem that I have to solve still and will be model specific :-) But I do seem to get reasonable answers right now.
So "64" is just in the end the endpoint’s maximum answer-label set. Most questions use only two or three of those rows. And, yeah, some calibration required. WIP on that
So I don't see the advantage to their approach until you're up beyond 6 or 7 questions?
Latest commits added gemma4 and instructions. I'll work on making a version of all of this that is standalone and not specific to DGX Spark.
Yeah, this is partially why in my approach I've done this instead, and not used diffusion model:
1. Convert the state into one shared prompt.
2. Run that shared prompt through the model once.
3. Fork the model’s internal state once per question.
4. Add a different question to each fork.
5. Ask each fork for its next-token scores.
6. Calculate only 64 possible label scores—not the whole vocabulary.
Basically ... skip decode.
Won't be as fast as doing diffusion model parallel across a pile of questions at once, but:
a) let's you use pretty much any existing text model (with some modifications). I've got qwen3.6 moe and qwen3.8 flash next running, am getting gemma4 working now
b) the problem you identified
It's possible I'm getting high on my own supply and misunderstand entirely the whole thing, but it seems to work?
https://github.com/rdaum/eider/commit/9c2d5c049068c33da2a48b...
I don't have the chutzpah to go creating PRs for vLLM to do the same.
If I understand it though it does mean you can evaluate a bunch of questions simultaneously, which is an advantage.
Also: While I think it's expected/normal to see LLM-generated programs... there's a lot of LLM written comments in that PR, which is sad to see. Auto-human.
why even bother with a network hop? build a specialized engine which does the prefill->measure cycle on local GPU/TPU/NPU with a model fine tuned for your application (e.g. gaming NPCs, autonomous driving, agricultural intelligence, drone.. target... selection, whatever)
the nice thing is that if you're skipping decode you're not as memory bandwidth bound.
https://www.reddit.com/r/LocalLLaMA/comments/1wijo3e/i_liter...
Not only is it replicable as you say, things like it already exist(ed).
The important bit of course is in the actual implementation: a) models fine tuned to produce good results for these types of questions and b) runtimes optimized to do this quickly and at scale
And prefill is way faster on GPU type hardware.
The thing is that the openjev stuff is a ... bit ... of a hack (a good one though):
It does this:
1. Send a throwaway request containing the shared state.
2. Hope SGLang keeps that text in its prefix cache.
3. Send a separate request for every question.
4. Each request repeats the shared beginning (but SGLang hopefully reuses the cached work in.)
5. Compute the complete vocabulary ; hundreds of thousands of possible tokens.
6. Keep only the few special answer tokens.
7. Convert those scores into probabilities.
Obviously this can all be done way more elegantly if you just own the inference engine -- fork / modify SGLang or vllm or llama.cpp, or do what I did in my bespoke inference engine (https://github.com/rdaum/eider/ commit https://github.com/rdaum/eider/commit/b2f981b7ebe0e338f60188...)
that ends up being, instead:
1. Convert the state into one shared prompt.
2. Run that shared prompt through the model once.
3. Fork the model’s internal state once per question.
4. Add a different question to each fork.
5. Ask each fork for its next-token scores.
6. Calculate only 64 possible label scores—not the whole vocabulary.
7. Convert the relevant scores into probabilities and return structured JSON.
I expect we'll see patches for llama.cpp and the others over the next few days/weeks and I also expect most model hosting providers will just end up providing this same service. I don't think Jev themselves have much of a moat. Though maybe it's more about their specific model and the training it gets.
I don't know how the Mac stuff compares on that front.
I have the same thing replicated in my own bespoke inference engine (for DGX Spark, in Rust & CUDA) and get answers pretty much as fast as the Jev openrouter endpoint.
https://github.com/rdaum/eider/
It's running over Qwen3.6. Getting it working with Qwen3.8 Flash Next now and getting a battery of tests and examples before I go more public with it.
That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.
It has also periodically aggressively helped subsidize, bankroll, and enforce the interests of some key sectors; notably the petroleum/energy sector. And finance.
In those sectors the state and private sector are fully intertwined in a strategic way.
I think "AI" is now joining that list. OpenAI and Anthropic will not be allowed to fall over or explode, and speculative investors I think are confident even with the dubious financial situation because they know this.
Especially insofar as there's now a strategic alignment of the fossil fuel sector and the "AI" datacentre sector as they are now becoming massive users of natural gas.
(Worse: Here in Canada that has taken on a very explicit role in that new datacentres seem to be pitched mainly in areas with remarkably traditionally expensive electricity and 100% reliance on natural gas [Alberta] and even coal [Saskatchewan] power generation -- instead of places like Quebec and B.C. that have copious hydroelectricity. On the surface it makes no sense until you realize it's more about finding customers for domestic natural gas than it is strategically about AI itself.)
Like Lockheed-Martin or Boeing, etc. there's just interpenetration between the corporate boardroom and the state. They act in each other's mutual interests.
The Chinese system is just more explicit and open about this.
And as a non-American, I can't trust the US state anymore than I can trust its dominant corporate entities. So I fail to see the advantage to the world to it being nationalized. In fact under the current administration this would be an even worse outcome.
That's more than Anthropic has done though.
And in that context NVIDIA had every motivation to keep their SDK somewhat abstracted higher up the chain and fully under their control and then be free to innovate in the lower bits. And this served them well as well as their customers.
My sense is that now with agent driven development this is basically evaporating. Agents are capable of at least prototyping/writing kernels for any hardware and ISA. e.g. OpenAI built their own custom hardware and ISA for it and then set agents loose on it writing kernels and claims great success. At least they're claiming this. And from my own experiences as a n00b entering this space, I can believe it.
TLDR I don't think vendor lock on the software side is going to work out for them as a strategy.
But luckily for them they continue to have really good hardware and good access to semiconductor fabrication. But just look at HotChips 2026 a couple weeks ago and look at the huge variety of new inference hardware coming down the pipe which looks completely unlike NVIDIA/CUDA.
What the hell is your problem? Getting on the Internet and hurling personal insets around. Get a grip.
I had LLMs write a pile of cuda-rust code and they were quite competent at it. Ported a bunch of (C++) CUDA kernels over, and ground away on them til they got equivalent performance
https://github.com/rdaum/eider/tree/main/backends/cuda-oxide
And mostly just DeepSeek 4.1 Flash, too. Not even a frontier model.
Sorry.
big hugs.
It's just a matter of degrees I guess.
I have good results with DS4.1 flash because I can iterate faster. I either provide it with correction, or it discovers its failures via the harness. And seems to respond well to empirical evidence rather than go in circles.
So it might need some prodding, but it's likely in this case it was able to brute force after several runs and collecting some evidence.