Janas: Run LLMs larger than RAM by streaming experts from SSD
github.com
github.com
I don't mean this negatively; open weights AI and open source tooling is about the only thing that gives me hope.
The fact that there are dozens of these things hints at the beginning of the dull normalisation of openly available, non-monopoly AI tools, which is what is needed if we are all to survive.
There may well be headroom within the software/architecture of LLMs but once you get down to streaming tokens, it is kind of running into the physics?
But also, there probably still is a lot of room for tuning and better utilizing the streaming capability, e.g. better speculative streaming to have the activated experts loaded in RAM by the time they're needed.
> Asianometry: High Bandwidth Flash Is Coming
I'd always assumed that the first generation of SSDs were really tested on consumers.
For comparison, here is the rough trajectory of price:
1978: $9.1 million/GB
1985: $920,000/GB
1990: $50,000/GB
1995: $1,000/GB [0]
2005: $100/GB
2006 is when the first commercial SSDs hit market in laptops, and by 2010 they were ~$3/GB, which brought them to wide adoption.
[0]: 2000 was a stagnant year, prices stayed the same from my research between ‘95 and the new millennium before dropping in 2005.
From what I've read one architecture is basically RAID0 at the flash chip/chiplet or PCB level, combining many fast flash chips into one wide channel and then turning that into a wide fast streaming storage system. Fast as in competitive with DRAM, at least for streaming a lot of data, which is what most AI inference wants.
Flash, even very fast flash, is much cheaper than DRAM per GB.
I'd like a specialized local inference node consisting of 2TB high bandwidth flash, 16-32GB of DRAM (for attention and other state), and a big TPU. That could run frontier size models in something the size of a large mini PC like an Intel Nuc. The lower cost of flash would probably move most of the cost to the processor(s).
I wonder how feasible it would be to DIY a board layout for basically RAID0 flash that's pretty good? A bunch of flash chips in parallel. Use something like a cheap fast ARM or RISC-V chip as the flash controller and have it do striping and present itself as an abstract drive over a thunderbolt or PCIe interface.
That's ~16 GB/s. Even if you do RAID0 of 4 such drives, that would get you 64GB/s. That's not a lot. Compare that to 1792 GB/s for RTX 5090: it's 100x faster than NVMe flash.
The work I've seen at places like Samsung and Intel is chip level.
Why hasn't this been done already? No market until recently. Existing flash is fast enough for SSD use cases, and making it fast enough to compete with DRAM makes it more expensive. Meanwhile it's not a true DRAM replacement because it's slow to write, slower than DRAM for true random access, and can wear out with many write cycles. It only makes sense for applications that are read-mostly, streaming, and require huge data, and that was a small market until AI.
Desktop CPUs support only one such slot (normally used for the GPU), together with another PCIe 5.0 M.2 and one independent PCIe 4.0 M.2. That sums to 88 GB/s for a consumer motherboard. A beefy CPU like an AMD 9950X should be sufficient for AI inference limited by the SSD throughput, even without a discrete GPU.
On a server MB, you may have e.g. 6 16-lane PCIe 5.0 slots, so you could use one slot for a discrete GPU and 5 slots for SSDs, for 320 GB/s throughput. At this point the price would be quite high even with the smallest available PCIe 5.0 SSDs, but it would still be low in comparison with the price of datacenter GPUs or of an equivalent amount of DRAM.
Well, I don't mean to sound rude or anything but that can be a lot to expect from maintainers of those projects.
As I said, it's better for projects like llama.cpp to be selective in this regard.
The benefit of llama.cpp etc is just ease of use for people not already familiar with the local AI landscape.
The speed obtained by this optimized program is many times higher than the current speed reached by llama.cpp when reading weights from an SSD.
It is unlikely that llama.cpp will include any time soon code capable of reading SSDs at the speed limited by hardware.
If the SSD files are memory-mapped, you need to provide at appropriate points in the program the correct madvise hints (the Linux-specific advices, not the useless POSIX advices) to free the unused memory and populate the memory that will be soon used, in such a way as to guarantee that there will never be any page faults while reading the weights.
Otherwise, you must used asynchronous I/O with O_DIRECT, as in this project, which requires a significant reorganization of the program in comparison with using memory-mapped files, to guarantee that the desired data is always available at the right time and the inference computation overlaps perfectly with reading continuously from the SSDs.
What a waste of time.
Strata already does this with Qwen3.8-Flash-Next: https://github.com/Niko1221/Strata
Are we hallucinating now too, not just the bots?
Only once? This has happened to me so many times with LLM-assisted development. I was writing an OCR engine and pathological shapes would cause OOM that would take the display out. On other occasions, the LLM would try to load 24GB of bf16 weights onto a 16GB GPU without understanding the consequences.
Have to encode this in logic: keep some space reserved for the display, and check before allocation. But some errors are bound to slip through.
---
Inspired by DwarfStar4 and Strata, I am trying to build my own gemma4-specific inference engine designed to run on gfx1201.
It is a lot of work. You cannot simply vibecode one in two days.
The goal is to get the 31B working on a 16GB card. Which means weights have to be < 4 bits. The 9070 XT has wmma support for int4. Which doubles compute performance on pp compared to i8/fp8. tg is limited by bandwidth.
Which seems sufficient for mortals who are not running gas town, but want a local model.
may I ask why?
oai luna runs at 60-70 toks/s; astra runs around 35-45 , are you just worried that your agentic work can't spread across a lot of parallel work?
Most of the tokens consumed by agents is reading stuff - docs, tool outputs, code etc. Previewing a content of a dumb single page of html is a few thousand tokens.
If you use claude or some other harness you can ask it how many cache writes/input tokens are there in avg session, and how much time it would take if preprocessing speed was just 60 tok/s.
But because prefill is parallelizable, datacenters will have an edge here for a while. For your one request, a datacenter may push 1.7kW power to a single machine to give you a result in a few seconds. The rest of the time machine will stay busy with other people’s traffic so it’s not a problem. For you, unless you’re a billionaire, you won’t have 1.7kW of processing power running idle 99.9% of time.
In 1-3 years we’ll have datacenters switch to dedicated chips that are way more energy efficient, and such chips will ultimately trickle down to PCs.
BTW - Apple is already introducing LLM specific architecture. They introduced Neural Accelerator (nit to be confused with Engine) that speeds up their GPUs 4x for LLM specific tasks. M5Ultra gets 100 tflops vs nvidia Rtx6000 getting 500tflops. „Just” 5x faster, but it’s beginning to look relevant.
Still 25 slower than a server farm B200. And when you ask Chat/Claude you get the full speed of B200.
What I’m waiting for is consumer dedicated LLM boards, a similar way we had soundcards and first gen 3d accels. No GPU, no 3D, purely LLM compute. I can see that happen in 2-3 years.
https://stealthgoat.com/deepswe-flash-next.html (Total AI pile). I benched this quant for 3 days on DeepSWE in a 16GB VRAM gaming PC. It held 90 t/s on avg I think. It’s solid. The authors claims of minimal intelligence loss hold. Sorry it renders bad in mobile will fix it soon.
It feels great to have a local model ripping at these speeds. I now use Flash NeXT as an active agent managing part of the engine build process and let it do some coding on easier chores :)
It doesn't look like a solved problem yet.
https://github.com/prabanta-dev/janas <- this, focuses on Qwen3-Next-80B-A3B.
https://github.com/Niko1221/Strata <- Qwen3.8-Flash-Next. I use this (tuned) and I cannot understate how satisfied I am.
https://github.com/antirez/ds4 <- DeepSeek 4 family + more.
There is no particular requirement for this to be limited to a single SSD. What happens when someone makes a GPU Card with enough SSDs attached to feed every RAM chip individually.
Given that you'd have probably a lot more capacity just to get to required bandwidth it could be used to switch very quickly between models (as all stored there would essentially be "loaded")
But the win here on servers is almost certainly power. High speed signals are power hungry, and the power scales with the length of the wire. HBF is designed to go in the same package as the GPU, so the distance is much shorter than with discrete drives.
e.g. each expert lives on one node of the network and the model gets streamed.
PS: There's heat you've got to worry about a bit, especially in very high speed drives, but that's about it (practically speaking). Bit rot prevention consumes some write cycles too but that is minuscule in comparison.
Maybe next you can run an LLM off of some old MFM drives
I don’t want to bother reading anything sub-Fable, and this one looks sub-Fable.