Comment by pjc50 | Hacker News Reader

Comment by pjc50 | Hacker News Reader

Parent Full thread

I'd look at this the other way round: everyone should now be aware that a computer is a series of processing units connected by network links whose latency is many times that of a register or cache access.

Some of those network links are labelled "PCIe" or "SATA", and some are labelled "Ethernet" or "DisplayPort", but they're all high-speed serial links. A PCIe lane is only 8Gbit, comparable to 10Gbit ethernet.

This means that your computer "backplane" design can be de-constrained; you don't have to draw a hard distinction between elements in the same case and those in the case of the machine next to you in the rack.

A rotating disk is roughly able to saturate a 1Gb ethernet link. Maybe we should just go full SAN and put Ethernet connectors on the drives.

Very true. Most servers are multiple processors (sockets) interconnected via a high speed network (QPI, HT), and each processor has links to its own memory and some PCI lanes. You could almost see that as multiple distinct computers on one board and in one case.

Hard disks with direct ethernet connections exist (f.e. Seagate Kintec).

Seagate Kinetic looks like an excellent idea, although horribly expensive.

It goes deeper than that. Some of those links are called QPI. Accessing memory on a multi-socket system can be much slower (40%) if the memory isn't connected to the socket you're on. In that case the request has to go to the other socket, then to RAM. In a quad-socket system, this might mean making two hops, first.

Even accessing cache in a multi-socket system sucks. Taking ownership of a cache line that's in another socket can cost 300 cycles. Ouch.

So in a way, programming current Intel NUMA systems is a lot more like writing distributed systems in the first place, if you care about performance. Yes, it's more reliable (your cross-socket read isn't gonna sporadically fail), but if you treat it as a single unit, things get slow.

To be fair, PCIe/SATA are a hell of a lot more reliable as "network links" as 10gb ethernet. SANs however, have the same failure modes as typical LANs.

>A rotating disk is roughly able to saturate a 1Gb ethernet link.

Locality is key there. Read randomly and you won't saturate a 100Mb link.

Reply on news.ycombinator.com