My understanding is that NVMe interface is pretty much as close as you can get to the CPU without being integrated. Is there a world where these things can operate as fast as RAM?
The best for consumer technology, yes. But future I/O protocols continue to improve and get better.
NVidia / IBM's collaboration on OpenCAPI (which is deployed in Summit as a CPU/GPU interface) has 300GBps I/O between the CPU and GPU, far faster than NVMe speeds (and even DDR4 RAM bandwidth).
And future chips may go even faster. I/O is probably one of the fastest growing aspects of a modern computer. PCIe 5.0, CXL, OpenCAPI, etc. etc. Lots and lots of new technology coming into play here.
There are even some products on making Flash-RAM work on the DDR4 interface. Non-volatile memory is what that's called. Intel's Optane works pretty well on that. Its not very mainstream but I hear reports that its impressive (slower than real RAM of course, but storage that has the bandwidth of RAM is still cool).
> Is there a world where these things can operate as fast as RAM?
Well... yesish. Flash is a kind of RAM (random access memory).
To answer your fundamental question though: No. Flash is fundamentally higher latency than DRAM (aka: DDR4). But with enough parallelism / big enough flash arrays (or DRAM arrays), you can continue to get higher and higher bandwidth.
At the top-of-the-line is SRAM, the RAM used to make L1 cache and registers inside of CPUs. This is very expensive and rarely used.
--------
Then you've got various non-mainstream RAMs: FRAM, MRAM, Optane, etc. etc.
The big thing about SSDs is that while sequential reads and writes have steadily improved, many workloads have not. From a cheap SSD to the best, there is very minimal difference in things like "how fast does my computer boot", "how fast does my game or game level load" and "how fast does visual studio load", or "how fast does gcc compile"
Partially, on a PC. The difference, on a PC, between a high end NVMe SSD and a SATA SSD is minimal for most use cases; small enough that an average user won't perceive much difference. The workloads in question (booting, loading a program, compiling code, etc.) involve a lot of operations that aren't bound by IO performance (network exchanges, decompressing textures, etc.) and haven't been optimized to maximize the benefit of high performance storage so the throughput and IOPS difference of the storage device don't entirely dominate.
I've got a workstation that has both SATA 3 and M.2 NVMe SSDs installed.
The SATA 3 device can do sustained reads of about 550 MB/sec, fairly close to the 600 MB/sec line rate of SATA 3.
The NVMe device can do about 1.3 GB/sec, faster than physically possible for SATA 3.
As the platter density increased, the head could glide over more data at the same spindle speed and you would get increasingly higher sequential speed. But you would not get much better latency - physically moving the head to a different track could not be done significantly faster. With each new generation, you would get only marginally better speed for many workloads that users actually care about. The latest HDDs could saturate SATA bus with sequential reads, but would still dip into KB territory (not MB, let alone GB) for sufficiently random-access workloads.
SSDs are similar in a sense that they can be massively parallelized for the increased throughput, but the latency of an individual cell is much harder to improve. Benchmarks will saturate the I/O queue and reach astronomical numbers by reading many cells in parallel, but for most desktop users, the queue depth of 1 (and the individual cell latency) is probably more relevant. That's why a lowly SATA SSD will be only marginally slower than the newest NVMe SSD for booting an OS or loading a game.
That said, it's still true that you typically won't see a very big difference. Any SSD can support loading a level in a few seconds. The difference between normal flash and ultra-fast flash won't show up much unless you're aggressively streaming level data off the drive in real time.
And that would likely tank the performance if the game is running from an HDD.
I suspect you could design the storage layer of your game/app/whatever to take into account the performance characteristics of the underlying drive, and adjust your I/O pattern to be more parallel or more serial accordingly.
I also suspect that would not be easy/economical to do within the time/budget constraints of most projects.
The name of the game is minimizing the head movements.
Try the following experiment: find a couple of multi-GB files on an HDD in order to copy them to an SSD.
First measurement: copy the first file then the second (then the third etc...) and sum-up the time.
Second measurement: just drag'n'drop all the files together and measure the total time.
Reboot before each measurement to purge the file cache.
There is serial as in reading byte 1 2 3 4. This is what you're talking about. This is good on any drive and great on hard drives.
There is serial as in read a byte, do a calculation, read a byte, do a calculation. This is what I was talking about. This is bad on any drive and even more noticeable on SSDs.
If you can have all your data in one single 500MB read, that's great. But I wasn't talking about that, because any drive can do a single giant read with good performance.
If you have to do 800 different reads, it's much better to ask for them all at once. You don't want to trickle them out one at a time. This is what my comparison was.
I've got a cheap SSD that will change your mind. It has amazingly poor performance though, I agree with the general concept that while quantitative differences can be measured, there's not a qualitative difference between competent SSDs.
That reminds me of non-volatile SRAM modules you can get by Microchip. How it works is that its just normal SRAM, but each cell is connected to a matching flash cell and when the mdoules voltage drops below a threshold, it writes the SRAM contents to flash in parallel. When the voltage is restored, it does the reverse. The user-observable effect is that you have normal SRAM (speed and it doesn't wear out) but its non-volatile. In reality, its a clever trick, but since the flash is only written when the module is powered off (not per-write to the module), you get a lot more use before it wears out.
Of course these modules are tiny (a few KB), but I still thought it was pretty interesting!
It seems to be kinda in between RAM and flash spec wise.
It wasn't really that fast, or wear resistant to replace flash.
Many customers try to use 3D XPoint as another part of the cache hierarchy in-between regular ssd and ram. It's actually pretty neat for faas workloads which want containers to be "warm" rather than "hot" or "cold"...