Hacker News Reader

Comment by mickg10 | Hacker News Reader

In reality - with decent switches at 25g - and no fec - node to node is reliably under 300ns (0.3 us)

Considering that 300 light-nanoseconds is about 90m, getting a response (or even just one-way) in that time is essentially running right at the limits of physics/causality.

Out of curiosity, how is that measured across machines?

(The first thing that comes to my mind would be to use an oscilloscope with two probes, one to each machine, but I’m guessing that’s not it.)

Measure the round trip and divide by two for the approximate one way time. It'd be really neat to measure the time it takes for a packet to travel in one direction, but it's somewhere between hard and impossible[1]; a very short path has less room to be asymetric though.

[1] If the clocks are synchronized, you can measure send time on one end, and receive time on the other. But synchronizing clocks involves estimating the time it takes for signals to pass im each direction, typically assuming each direction takes half the round trip.

You can use something like White Rabbit (https://en.wikipedia.org/wiki/White_Rabbit_Project) to keep clocks in sync. That still involves estimates, but a dedicated time sync network can do things like make sure all the cables are the same length.

Copper white rabbit is special, it uses the same wire in both directions (1000BASE-T phy with added carrier phase lock to and from outside clocks).

What is the hardware involved in this test?

Is this is with PCIe based Ethernet NICs? A single PCIe read probably takes at least a hundred nanos, right?

Maybe I could see it with something like a busy poll pingpong over RDMA/Infiniband, but that seems really low for “traditional” networking. It’s probably not possible to even send and receive a packet via loopback device that fast.

Reply on news.ycombinator.com