Well, all memory operations must be translated by an MMU. So can the MMU be a distributed beast that lives in the CPU, the memory and the IO?
That "blah = blah->next" memory operation could be:
* Going to swap thanks to swap / overcommit behavior on Linux/Windows.
* Or going to a file thanks to mmap
* Going to Ethernet and to another computer thanks to RDMA
So... no. I'm pretty sure the CPU is the proper place for that kind of routing.
moreover, if we have support for mmap at the MMU level, we can cut the CPU bottleneck for disk access entirely. the disk controller can already handle DMA, but there's simply no way for things that aren't the CPU to trigger it. DirectStorage is an effort for GPUs to trigger it, but what if we could also trigger it by other means?
You have a linked list. Each memory location, since it’s user-space, stores the virtual address to the next location. How do you offload a program to process this “in situ”? You’d need to translate these to physical addresses. Parent is 100% correct about the challenge this poses.
* MMAP is still on the table: different processes can share RAM at different addresses. (Process#1 thinks the data is at memory location 0x90000000, Process#2 thinks the data is at 0x70000000, but in both cases, the data is at physical location 0x42).
* Physical location 0x42 is a far-read on a far-away NUMA node. Which means the CPU#0 now needs to send a message to a CPU#1 very far away to get a copy of that RAM. This message traverses Intel Ultrapath Interconnect or AMD Infinity fabric (proprietary details), but its a remote message that happens nonetheless.
* Turns out CPU#1 has modified location 0x42. Now CPU#1 must push the most recent copy out of L1 cache, into L2 cache... then into L3 cache, and then send it back to CPU#0. CPU#0 has to wait until this process is done. If CPU#1 wants to modify the data again (or even read it), it may require messages from CPU#0 (who is now the owner of the data, according to simple MESI models).
Modern computers work very hard to hold the illusion of a singular memory space. Eventually, these details are turned into a consistent memory model and become well-ordered sequential operations. The CPU is a good place for that.
---------------
That's how stuff works _today_. If you wanted to make a new programming model that's incompatible, that's fine. (CUDA does it: GPUs don't have as many virtual-memory features as a CPU. And __shared__ memory has a different model than L1 cache.)
But if you invent a new memory model that does things differently, it means that it won't work for the vast majority of code. Which means you need to bootstrap a new programming environment (much like how CUDA bootstrapped a new community from scratch).
I don’t disagree that the memory model becomes more complex. For one CPU cache invalidation becomes really tricky. So do memory coherence rules.
I’m less clear how mmap matters here. That’s just a mechanism the OS uses to hand out views into the page cache to the process - if you’ve solve the virtual-physical mapping (which you have to do) then mmap is not relevant.
You’re spot on though that the particular design decisions are critical for this to be successful - pick the wrong point on the complexity/cost/perf curve and your solution will definitely be DOA.