HNHacker News
TopNewBestAskShowJobs

cmrdporcupine

23,652 karma · joined December 5, 2014

Surfin' the Dunning-Kruger wave since the 1990s.

Member of Technical Staff, AI @ Verda https://verda.com/

some src @ https://github.com/rdaum/

submissionscomments
cmrdporcupine··on Qwen Image 2.1
Everything is combined and uneven, including the opinions of hackernews commenters?

There is no single opinion, and clearly no single Chinese approach.

Also Chinese labs are in particular very careful about anything which be used to create pornographic content, which is highly illegal in the PRC.

cmrdporcupine··on I built non-autoregressive decision models with RL a year ago
I'll just say that even though I was poor and without a job and living on unemployment insurance for a year...

The implosion of hype after the .com crash was actually kind of a ... relief.

cmrdporcupine··on I built non-autoregressive decision models with RL a year ago
Jev is only on people's mouths because they made friends with venture capitalists and used the publicity blowhorns that come with that.

Whereas the other guy went through the unglorious but formerly respectable path of publishing software and papers for other professionals to look at. A year ago.

We're in a bad place where the latter looks less reliable than the former.

(EDIT: I'm not saying the research here is in fact the same as what "Jev" is doing; and Jev is in fact more "product shaped." But I think it's important to temper the hype and back up and focus on the fact that this whole industry is built on research by both academics and enthusiasts ... first ... and gold rushes can often bulldoze over those people who are focused primarily on making-doing-researching instead of fundraising-hyping-promoting. That's not good.)

cmrdporcupine··on OpenJev
Ok I tried it. And it's faster. Though I'm still in verifying quality phase.

But only works for Gemma4, or other attention-only models. (i.e. not the Qwen3.8 models I had working with the other way)

I'm getting 63ms per answer for 1 question, 79 for 3, 93.7ms for 5. So scales nicely, too. ~55ms fixed cost + ~7-8ms per additional question.

Much better. Thanks. I'll commit and share work after testing it some more.

EDIT: how can I credit you in the commit body?

cmrdporcupine··on OpenJev
I think you're right. Better. I will have to think through if it would be faster or slower. If understand what you're getting at with this.. broken analogy...

What I described was -- we have a bunch of orders to the kitchen, all of which have the same "base" meal but different topics.

> Cook the "base" meal once, divide servings onto multiple plates, then add different toppings to each plate.

vs what you suggest:

> Put the base meal and every topping through the kitchen together, but use some kind of dividers so the toppings never mix up together.

Except.. ok, that analogy is confusing lol.

cmrdporcupine··on How OpenAI Used Its Own LLMs to Design Its Jalapeño Chip
Or they'll just build a competitor in RISC-V instead and that's fine.

Except the problem is not restricted to the actual ISA or its HDL implementation, etc.

It's even just getting space / time in a fab at that advanced of a process node.

cmrdporcupine··on Claude Code now reads AGENTS.md if there is no Claude.md
Only took them a year and a half of everyone complaining to finally do the right thing.

Congrats.

cmrdporcupine··on OpenJev
So the model normally (like during a normal decode) takes its final hidden vector and multiplies it by the entire vocabulary head -- so like about 250K rows -- to produce one logit per possible next token (and then so on and so on...)

Instead I use a fixed set of up to only 64 single-token labels. At model load, I gather only those 64 rows from the vocabulary head into a small matrix. Each question maps its permitted answers onto some of those labels.

It is a probability distribution conditional on the allowed labels. Calibration is a separate problem that I have to solve still and will be model specific :-) But I do seem to get reasonable answers right now.

So "64" is just in the end the endpoint’s maximum answer-label set. Most questions use only two or three of those rows. And, yeah, some calibration required. WIP on that

cmrdporcupine··on OpenJev
fwiw, w/ gemma4 -- non-diffusion -- I get about 170ms for a single question -> answer and then an additional ~33ms on adding more. While I see people reporting 300ms for this vLLM PR on same hardware (Spark.)

So I don't see the advantage to their approach until you're up beyond 6 or 7 questions?

Latest commits added gemma4 and instructions. I'll work on making a version of all of this that is standalone and not specific to DGX Spark.

cmrdporcupine··on OpenJev
> It seems by sharing KV cache all questions will be visible

Yeah, this is partially why in my approach I've done this instead, and not used diffusion model:

1. Convert the state into one shared prompt.

2. Run that shared prompt through the model once.

3. Fork the model’s internal state once per question.

4. Add a different question to each fork.

5. Ask each fork for its next-token scores.

6. Calculate only 64 possible label scores—not the whole vocabulary.

Basically ... skip decode.

Won't be as fast as doing diffusion model parallel across a pile of questions at once, but:

a) let's you use pretty much any existing text model (with some modifications). I've got qwen3.6 moe and qwen3.8 flash next running, am getting gemma4 working now

b) the problem you identified

It's possible I'm getting high on my own supply and misunderstand entirely the whole thing, but it seems to work?

https://github.com/rdaum/eider/commit/9c2d5c049068c33da2a48b...

I don't have the chutzpah to go creating PRs for vLLM to do the same.

cmrdporcupine··on OpenJev
This PR is interesting but it's making the assumption that what Jev has done is based on a diffusion model or that a diffusion model is superior for this work. Which may or may not be the case.

If I understand it though it does mean you can evaluate a bunch of questions simultaneously, which is an advantage.

Also: While I think it's expected/normal to see LLM-generated programs... there's a lot of LLM written comments in that PR, which is sad to see. Auto-human.

cmrdporcupine··on OpenJev
and frankly for many of the kind of thing people probably want to use this for... you would want to run locally anyways.

why even bother with a network hop? build a specialized engine which does the prefill->measure cycle on local GPU/TPU/NPU with a model fine tuned for your application (e.g. gaming NPCs, autonomous driving, agricultural intelligence, drone.. target... selection, whatever)

the nice thing is that if you're skipping decode you're not as memory bandwidth bound.

cmrdporcupine··on OpenJev
the models will come. or be fine tuned
cmrdporcupine··on OpenJev
There's also prior art. Or probably, anyways.

https://www.reddit.com/r/LocalLLaMA/comments/1wijo3e/i_liter...

Not only is it replicable as you say, things like it already exist(ed).

The important bit of course is in the actual implementation: a) models fine tuned to produce good results for these types of questions and b) runtimes optimized to do this quickly and at scale

cmrdporcupine··on OpenJev
Basically it's: skip decode, just do prefill then do some measurements. That's the crude description anyways.

And prefill is way faster on GPU type hardware.

cmrdporcupine··on OpenJev
It's good people moved this quickly on this stuff.

The thing is that the openjev stuff is a ... bit ... of a hack (a good one though):

It does this:

1. Send a throwaway request containing the shared state.

2. Hope SGLang keeps that text in its prefix cache.

3. Send a separate request for every question.

4. Each request repeats the shared beginning (but SGLang hopefully reuses the cached work in.)

5. Compute the complete vocabulary ; hundreds of thousands of possible tokens.

6. Keep only the few special answer tokens.

7. Convert those scores into probabilities.

Obviously this can all be done way more elegantly if you just own the inference engine -- fork / modify SGLang or vllm or llama.cpp, or do what I did in my bespoke inference engine (https://github.com/rdaum/eider/ commit https://github.com/rdaum/eider/commit/b2f981b7ebe0e338f60188...)

that ends up being, instead:

1. Convert the state into one shared prompt.

2. Run that shared prompt through the model once.

3. Fork the model’s internal state once per question.

4. Add a different question to each fork.

5. Ask each fork for its next-token scores.

6. Calculate only 64 possible label scores—not the whole vocabulary.

7. Convert the relevant scores into probabilities and return structured JSON.

I expect we'll see patches for llama.cpp and the others over the next few days/weeks and I also expect most model hosting providers will just end up providing this same service. I don't think Jev themselves have much of a moat. Though maybe it's more about their specific model and the training it gets.

cmrdporcupine··on OpenJev
Performance for this kind of thing should be best on any hardware that has high prefill speeds. As basically this is "do prefill only, measure scores, skip decode entirely".

I don't know how the Mac stuff compares on that front.

I have the same thing replicated in my own bespoke inference engine (for DGX Spark, in Rust & CUDA) and get answers pretty much as fast as the Jev openrouter endpoint.

https://github.com/rdaum/eider/

It's running over Qwen3.6. Getting it working with Qwen3.8 Flash Next now and getting a battery of tests and examples before I go more public with it.

cmrdporcupine··on Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint
What are you getting for prefill?
cmrdporcupine··on Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint
What I'd love to see is this done for DS4.1 Flash.

That would bring it down to the point where it can fit in 128GB on things like the Spark or Strix Halo.

cmrdporcupine··on OpenAI's Misalignment Framework: A Tactical Bid to Preempt Global AI Governance
Beyond the defense sector, the US state has always intervened publicly and privately to mediate and balance competing corporate "private" interests.

It has also periodically aggressively helped subsidize, bankroll, and enforce the interests of some key sectors; notably the petroleum/energy sector. And finance.

In those sectors the state and private sector are fully intertwined in a strategic way.

I think "AI" is now joining that list. OpenAI and Anthropic will not be allowed to fall over or explode, and speculative investors I think are confident even with the dubious financial situation because they know this.

Especially insofar as there's now a strategic alignment of the fossil fuel sector and the "AI" datacentre sector as they are now becoming massive users of natural gas.

(Worse: Here in Canada that has taken on a very explicit role in that new datacentres seem to be pitched mainly in areas with remarkably traditionally expensive electricity and 100% reliance on natural gas [Alberta] and even coal [Saskatchewan] power generation -- instead of places like Quebec and B.C. that have copious hydroelectricity. On the surface it makes no sense until you realize it's more about finding customers for domestic natural gas than it is strategically about AI itself.)

cmrdporcupine··on OpenAI's Misalignment Framework: A Tactical Bid to Preempt Global AI Governance
The nature of the US state is such that the distinction between nationalized and not is almost meaningless.

Like Lockheed-Martin or Boeing, etc. there's just interpenetration between the corporate boardroom and the state. They act in each other's mutual interests.

The Chinese system is just more explicit and open about this.

And as a non-American, I can't trust the US state anymore than I can trust its dominant corporate entities. So I fail to see the advantage to the world to it being nationalized. In fact under the current administration this would be an even worse outcome.

cmrdporcupine··on How GLM built its own inference infrastructure
It comforts Americans to believe their exceptionalism is both persistent/eternal and fully justified.
cmrdporcupine··on How GLM built its own inference infrastructure
OpenAI did recently get into how they had been building their own hardware and doing RSI with it.

That's more than Anthropic has done though.

cmrdporcupine··on Nvidia announces native GPU programming in Rust
My thoughts on this, as a person who has recently coming around to working in this space is that up to now the convenience and "simplicity" of working in CUDA as it is has been a giant moat for NVIDIA. Having a whole toolchain with a C++ dialect and a giant extant pile of code out there that looked familiar to people meant they've "won" the AI wars.

And in that context NVIDIA had every motivation to keep their SDK somewhat abstracted higher up the chain and fully under their control and then be free to innovate in the lower bits. And this served them well as well as their customers.

My sense is that now with agent driven development this is basically evaporating. Agents are capable of at least prototyping/writing kernels for any hardware and ISA. e.g. OpenAI built their own custom hardware and ISA for it and then set agents loose on it writing kernels and claims great success. At least they're claiming this. And from my own experiences as a n00b entering this space, I can believe it.

TLDR I don't think vendor lock on the software side is going to work out for them as a strategy.

But luckily for them they continue to have really good hardware and good access to semiconductor fabrication. But just look at HotChips 2026 a couple weeks ago and look at the huge variety of new inference hardware coming down the pipe which looks completely unlike NVIDIA/CUDA.

cmrdporcupine··on Nvidia announces native GPU programming in Rust
Chill man, holy crap. I never told him not to learn. I was speaking directly to his "good thing LLMs don't know how to use this yet".

What the hell is your problem? Getting on the Internet and hurling personal insets around. Get a grip.

cmrdporcupine··on Nvidia announces native GPU programming in Rust
Sad to break it to you, but...

I had LLMs write a pile of cuda-rust code and they were quite competent at it. Ported a bunch of (C++) CUDA kernels over, and ground away on them til they got equivalent performance

https://github.com/rdaum/eider/tree/main/backends/cuda-oxide

And mostly just DeepSeek 4.1 Flash, too. Not even a frontier model.

Sorry.

cmrdporcupine··on Forgejo <=16.0.3 Critical RCE
i love you too, buddy.

big hugs.

cmrdporcupine··on EU chief opens door for Canada to become 'associate member'
There are no good guys. The foxes are in charge of the hen house in every country. E.g. Carney is using all of this as a chance to completely gut every climate change initiative possible and slash corporate taxes, shore up existing Canadian oligopolies etc. etc. The nationalism is a drug which can be used by elites in all places for their own purposes.

It's just a matter of degrees I guess.

cmrdporcupine··on DeepSeek v4.1 Flash Is Now Our Best Hacking Model
Thing is that GLM 5.3 is many multiples the cost to run, and slower.

I have good results with DS4.1 flash because I can iterate faster. I either provide it with correction, or it discovers its failures via the harness. And seems to respond well to empirical evidence rather than go in circles.

So it might need some prodding, but it's likely in this case it was able to brute force after several runs and collecting some evidence.

cmrdporcupine··on Dream-RSI: Recursive Self-Improvement through Evolving Worlds
That's my point about the harness.
← PreviousPage 6 of 34Next →