Hacker News Reader

Comment by blibble | Hacker News Reader

> While latency over a local network connection is low, it's nowhere near as low as over a PCI or SATA connection that spans less than a meter. This means that the average latency of disk operations (from the perspective of the operating system) can be on the order of a couple milliseconds, compared to half a millisecond for directly-attached disks.

uh, you can get a packet from your software on one machine, through a switch and into your software on the second machine in 1-2us if you know what you're doing

(of course, that's without any cloud bullshit like passing the packet through half a dozen levels of virtualisation, then through software defined networks)

View on HN

The other replies are assuming networking in a big network is inherently slower than in a small network. I used to work at Google in Tech Infra, so I'll offer an alternate perspective while trying not to spill secrets.

First, Google has enough money that they can build their entire network out of custom hardware, custom firmware, and patch the kernel + userspace. A datacenter at Google scale is architecturally similar to a supercomputer cluster running on InfiniBand. You will never be able to replicate the performance of Google's network by buying some rackmounted servers from Dell and plumbing them together with Cisco switches.

Second, assuming a reasonably competent design, adding more machines to a network doesn't significantly increase the latency of that network. You'll see better latencies between machines in the same rack than between two racks, but this is a matter of single microseconds rather than milliseconds. Additional latency from intermediate switches is measured in nanoseconds.

Third, Google publishes an SLA on round-trip network latency between customer VMs at <https://cloud.google.com/vpc/docs/vpc>. Their "tail latencies less than 80μs at the 99th percentile" translates to ~40μs for one way, and honestly for customer VMs a lot of that happens in the customer kernel + virtualization layer. A process running on bare-metal, such as a kernel reading a remote network block device, can (IIRC) expect single-microsecond latencies to get one packet onto a nearby machine.

> The other replies are assuming networking in a big network is inherently slower than in a small network.

well yes, not true with modern switches that support cut-through forwarding

it's super-common in our space to bypass the kernel entirely, writing into the NIC buffers directly with prepared packet headers, and the card has pushed part of the packet out onto the wire, through switches and into the target machine's NIC buffers before it's even finished being written

typical "SLA"s are 0 packets dropped during a session, where a single drop raises an alert that is then investigated

> You will never be able to replicate the performance of Google's network by buying some rackmounted servers from Dell and plumbing them together with Cisco switches.

and yet, somehow we do quite a bit better (admittedly they are very, very expensive switches)

I get that people that work at Google like to think they're working on problems more advanced than those of mere mortals, but with the latencies you've described we'd be out of business several times over

(not to mention none of the clouds support multicast)

I take it you work in trading?

That's just different niche. I assume that you work in trading based on the word "multicast".

What Google needs is "Big+Cheaper" datacenters, and it has to work with codes written by 100000 different mere morals. What you described is in the "Small+Expensive" field, but with extreme worst-case performance demand.

"Big+Expensive" = Supercomputer "Small+Cheaper" = ??? (Note that the Big+Cheaper solutions not necessarily work for this, as you can't amortize and ignore one-time R&D/ops cost anymore)

I think you may have misread my comment.

The other replies are accepting multi-millisecond latencies as a given, and think that Google's network must be slower than even a basic copper-wired LAN because it's bigger.

My response is something like "just because the network's bigger doesn't mean it's slower".

  >> You will never be able to replicate the performance of Google's network
  >> by buying some rackmounted servers from Dell and plumbing them together
  >> with Cisco switches.
  >
  > and yet, somehow we do quite a bit better (admittedly they are very,
  > very expensive switches)

With respect, if you're in the trading business, your network almost certainly contains custom hardware. I bet it looks a lot closer to Google's than it does to the guy plugging cat5e into a Dell.

> uh, you can get a packet from your software on one machine, through a switch and into your software on the second machine in 1-2us if you know what you're doing

This isn’t a pair of servers sitting in a room with a switch in between them on a simple /24 subnet. It’s a gigantic cloud data center with a massive network spanning a huge number of devices.

The simple things you can get to work in a small setup don’t apply at this scale.

the example I gave was a simplified (though true) situation of a system local bus

it's common to have a dedicated trading network completely parallel and isolated from the main DC network for each upstream connection, of which there are dozens

and these are (at least) double redundant through the entire network, typically using redundant packet arbitration to near guarantee zero drops over a session

Google's network is super-dynamic and flexible, but slow and likely over-contended

ours is the exact opposite

> that's without any cloud bullshit like passing the packet through half a dozen levels of virtualisation, then through software defined networks

I mean, by definition, a public cloud wouldn’t work without layers of virtualization and SDN.

while i am not a "cloud expert", i'm pretty sure that google datacenters have more than one switch. furthermore, while the precise datacenter layout is probably secret, i infer that the the compute servers and the data servers cannot be housed in the same or nearby racks, since both are independently provisionable and can be arbitrarily attached. for example, if i start a vm and connect a drive, even if google could put them right next to each other in the datacenter, what happens if i start another vm, which for fragmentation reasons gets placed way on the other side of the building, detach the drive and connect it to this new one?

therefore, there must at least be several switches between the devices, and realistically, probably one or more routers.

it was an example

with modern switches it barely matters how many they are in between the machines

Reply on news.ycombinator.com