Hardware is becoming the choke-point. If that gets resolved, things will start to change even faster
From what I've read one architecture is basically RAID0 at the flash chip/chiplet or PCB level, combining many fast flash chips into one wide channel and then turning that into a wide fast streaming storage system. Fast as in competitive with DRAM, at least for streaming a lot of data, which is what most AI inference wants.
Flash, even very fast flash, is much cheaper than DRAM per GB.
I'd like a specialized local inference node consisting of 2TB high bandwidth flash, 16-32GB of DRAM (for attention and other state), and a big TPU. That could run frontier size models in something the size of a large mini PC like an Intel Nuc. The lower cost of flash would probably move most of the cost to the processor(s).
I wonder how feasible it would be to DIY a board layout for basically RAID0 flash that's pretty good? A bunch of flash chips in parallel. Use something like a cheap fast ARM or RISC-V chip as the flash controller and have it do striping and present itself as an abstract drive over a thunderbolt or PCIe interface.
That's ~16 GB/s. Even if you do RAID0 of 4 such drives, that would get you 64GB/s. That's not a lot. Compare that to 1792 GB/s for RTX 5090: it's 100x faster than NVMe flash.
The work I've seen at places like Samsung and Intel is chip level.
Why hasn't this been done already? No market until recently. Existing flash is fast enough for SSD use cases, and making it fast enough to compete with DRAM makes it more expensive. Meanwhile it's not a true DRAM replacement because it's slow to write, slower than DRAM for true random access, and can wear out with many write cycles. It only makes sense for applications that are read-mostly, streaming, and require huge data, and that was a small market until AI.
Desktop CPUs support only one such slot (normally used for the GPU), together with another PCIe 5.0 M.2 and one independent PCIe 4.0 M.2. That sums to 88 GB/s for a consumer motherboard. A beefy CPU like an AMD 9950X should be sufficient for AI inference limited by the SSD throughput, even without a discrete GPU.
On a server MB, you may have e.g. 6 16-lane PCIe 5.0 slots, so you could use one slot for a discrete GPU and 5 slots for SSDs, for 320 GB/s throughput. At this point the price would be quite high even with the smallest available PCIe 5.0 SSDs, but it would still be low in comparison with the price of datacenter GPUs or of an equivalent amount of DRAM.
There may well be headroom within the software/architecture of LLMs but once you get down to streaming tokens, it is kind of running into the physics?
But also, there probably still is a lot of room for tuning and better utilizing the streaming capability, e.g. better speculative streaming to have the activated experts loaded in RAM by the time they're needed.
> Asianometry: High Bandwidth Flash Is Coming
I'd always assumed that the first generation of SSDs were really tested on consumers.
For comparison, here is the rough trajectory of price:
1978: $9.1 million/GB
1985: $920,000/GB
1990: $50,000/GB
1995: $1,000/GB [0]
2005: $100/GB
2006 is when the first commercial SSDs hit market in laptops, and by 2010 they were ~$3/GB, which brought them to wide adoption.
[0]: 2000 was a stagnant year, prices stayed the same from my research between ‘95 and the new millennium before dropping in 2005.