Currently, the fastest way to connect flash is via PCIe 5.0 x4. Right?
That's ~16 GB/s. Even if you do RAID0 of 4 such drives, that would get you 64GB/s. That's not a lot. Compare that to 1792 GB/s for RTX 5090: it's 100x faster than NVMe flash.
That's ~16 GB/s. Even if you do RAID0 of 4 such drives, that would get you 64GB/s. That's not a lot. Compare that to 1792 GB/s for RTX 5090: it's 100x faster than NVMe flash.
Desktop CPUs support only one such slot (normally used for the GPU), together with another PCIe 5.0 M.2 and one independent PCIe 4.0 M.2. That sums to 88 GB/s for a consumer motherboard. A beefy CPU like an AMD 9950X should be sufficient for AI inference limited by the SSD throughput, even without a discrete GPU.
On a server MB, you may have e.g. 6 16-lane PCIe 5.0 slots, so you could use one slot for a discrete GPU and 5 slots for SSDs, for 320 GB/s throughput. At this point the price would be quite high even with the smallest available PCIe 5.0 SSDs, but it would still be low in comparison with the price of datacenter GPUs or of an equivalent amount of DRAM.
The work I've seen at places like Samsung and Intel is chip level.
Why hasn't this been done already? No market until recently. Existing flash is fast enough for SSD use cases, and making it fast enough to compete with DRAM makes it more expensive. Meanwhile it's not a true DRAM replacement because it's slow to write, slower than DRAM for true random access, and can wear out with many write cycles. It only makes sense for applications that are read-mostly, streaming, and require huge data, and that was a small market until AI.