Cache coherency is optimistic, message passing is pessimistic.
Cache coherence protocol are all designed to keep everybody in sync. So if you scale to thousands of cores, you better sync selectively and application specific. Thus it makes sense to use message passing (actor model).
It is unclear where the it makes sense to switch. Somewhere between 2 and 1000.
Hybrids are also an option. Supercomputers often use OpenMP (shared memory) and MPI (message passing) in a two level way.
Not some, almost all supercomputer nowadays use an hybrid model.
For three main reasons:
- Memory is a precious resource and separated process space + messaging is more memory hungry than a shared hybrid process space
- Hybrid machines with CPU + GPU on every node are everywhere and they requires a shared memory model.
- Debugging a pure MPI model at scale is a huge pain in the ass that take some of the most complex debugger in the planet.
They don't?
In practice cache coherency is a correctness mechanism; actually using shared memory concurrently R/W is problematic, performance wise. That's how we get scatter/gather-style systems, specifically to avoid concurrent R/W.
Wait, what? I'm looking at an image of an unlidded Zen 2 CPU right now and it's a bunch of discrete compute devices surrounding a switch. Inside these devices are multilevel caches, just like real world network systems. The central "chiplet" multiplexes access to external buses over multiple serial PCI lanes; any network engineer would would recognize this pattern as trunking. All of this looks exactly like a network.