Hacker News Reader

Comment by pixelmonkey | Hacker News Reader

The thing I learned that blew me away is that with their new networking stack, the cross-AZ communication within a region is "always <2ms latency and usually <1ms".

As mentioned in the slides, that's the same latency usually associated with SSDs and 100X better than the latency typically associated with cross-region networking. This suggests to me that running distributed databases in multiple AZ's has almost no latency penalty (e.g. you won't even paying eventual consistency / replication lag taxes that you might think are a danger). That's pretty damn cool.

View on HN

I was a big Xen guy back in the day.

The big takeaway from Hamilton's talk was actually something that dawned on me when they released the c3/m3/i2 instances which is what they have done with SR-IOV.

The software stack that implements the AWS VPC fast path must be running on the NIC itself. This is a big deal. Being able to implement the require isolation/tunneling/encapsulation logic and routing in the NIC is a huge performance boost and drastically simplifies the hypervisor. Well, the software portion of the hypervisor atleast.

If you listened closely to the talk he laid out the differences in latency impact of each layer in the networking stack, from the fibre (nanoseconds) up to the software stack at some number of milliseconds, several orders of magnitude worse than anything else in the stack.

The reason why the quoted latency is so large is how para-virtualized networking is implemented. In order for it to be (bandwith) performant it needs to use large ring buffers with many segments. This is very throughput optimised and latency suffers a ton as a result.

By moving all of the queuing to the NIC you get a bunch of benefits, namely the dom0 (basically the host in Xen terminology) is no longer involved in pushing packets and you are not incuring the cost of the Linux networking stack 2x. In the paravirt model the skbs are transferred across the circular buffer into the host OS where they are injected into a virtual interface, thus traversing the full net stack again.

In the SR-IOV model the address space of the virtual NIC is mapped into the guest OS using Intel's IO-MMU extensions and the guest is then able to communicate directly with the NIC, thus 100% bare metal performance.

If SR-IOV was the only improvement it would be impressive, however it's the consequence of it's existence which makes the biggest difference. If the guest is talking directly to the NIC, then all of the encap/decap is HW accelerated too and in theory this means the full networking stack is end-to-end in HW.

Note: standard out-of-the-box SR-IOV allows VLAN tagging/stripping outside of the guest control, maybe they are simply using this in conjunction with a layer 3 switch to handle the VPC stuff?

Amazon VPC is far more complex: https://www.youtube.com/watch?v=Zd5hsL-JNY4 (excellent talk from one of the creators of VPC)

I just watched that youtube video.. I had thought that VPC's were a clever combination of xvlan or grsec tunnels; I had no idea how custom it was. That's crazy in combination with sr-iov as you pointed out. I gather they are either running the VPC stuff on the card or what it's connected to.

I'd guess they are using VXLAN or another similar technology.

Probably something closer to MPLS all things considered. If you think about it they effectively are a carrier network with many virtual customer networks on top.

MPLS has a bunch of advantages over VXLAN and is decidedly L3, which is more suited to AWS VPC. The real question however is what the control plane looks like.

They may just go pure carrier style and just do tagging and resource control on the NIC but it's entirely possible for the NIC to be doing much more, especially if they have a dynamic control plane similar to OpenFlow (though probably not exactly OpenFlow.. it's less useful than it sounds).

I would love to see a talk on AWS VPC internals rather than sitting around taking pot shots. :)

We see this using FoundationDB across 3 AZs. Absolutely not a problem at all with great performance and reliability.

> This suggests to me that running distributed databases in multiple AZ's has almost no latency penalty (e.g. you won't even paying eventual consistency / replication lag taxes that you might think are a danger).

Yes, Hamilton points that out in his talk: https://www.youtube.com/watch?v=JIQETrFC_SQ

Indeed. It would be interesting to know the distance between two AZs, provided 2ms is the RTT. And considering many factors including - switching and network latency including encryption / decryption on cross AZ traffic (if they do)

> The above shows the US East region in Ashburn, Virginia. It has five availability zones, and these are protected areas that are isolated from each other by a couple of kilometers ...

I wonder what it would take to get it further towards the speed-of-light limit.

2 ms for, say, 3 km is 1500 km/s, or around 0.75% the speed of light in fiber (~200,000 km/s).

I wonder what the bottleneck is.

Reply on news.ycombinator.com