Hacker News Reader

Comment by Cyph0n | Hacker News Reader

> handle the encryption of every packet passing between servers within and across vast data centres

This sounds like it’s doing packet processing in software, which doesn’t seem scalable, especially for the traffic volume I would assume Apple handles. Anyone have a clue what kind of traffic volume and bandwidth we’re looking at here?

Granted, I might be overestimating the requirements given the industry I work in (service provider routing).

View on HN

None of the big clouds use hardware network encryption. All of the resources in your network are at the leaves, so you might as well use their CPUs to do it. I think Google published that crypto and compression in their internal RPC stack cost them less than 1% of their global CPU resources.

> I think Google published that crypto and compression in their internal RPC stack cost them less than 1% of their global CPU resources.

Not sure if this makes your point compelling. At Google's scale, if you were able to reduce global CPU usage by 0.1%, that would probably be a massive win.

Reduce it how? By deploying expensive, unreliable, and hard-to-manage vendor crypto garbage? That is undoubtedly not a trade that a Google would make.

You reduce it by using custom routing ASICs that can handle orders of magnitude more traffic than a CPU with equal or lower power consumption. To put things into perspective: Cisco’s latest routing ASIC — the Cisco Silicon One — can handle 10.8 Tbps (in fixed mode).

I’m sure Google and others have evaluated this, but it’s just kind of surprising that they opt to do per-packet processing in software.

Disclaimer: I write router software at Cisco.

10tb sounds like a big scary number but when you think about it that's only 100 100gb ports. It should be no surprise to you that a company making their own switches and NICs, where every RPC is individually authenticated and possibly also encrypted with session keys specific to the peer has no need of vendor "acceleration".

https://cloud.google.com/blog/products/networking/google-clo...

https://cloud.google.com/security/encryption-in-transit/appl...

10 Tbps is not a scary number? I am honestly still in awe of how such a small box can route an entire city’s worth of traffic. Different perspectives I suppose.

Regarding your points: I am not sure I completely follow.

Firstly, as far as I know, Google does not make its own switching or routing ASICs.

Secondly, virtually all switching and routing ASICs are highly programmable. So if you need a custom protocol, you can implement it using the vendor’s microcode (e.g., P4). In other words, you are not limited to the protocols that the vendor has implemented.

Given the above, I don’t see what kind of technical requirements Google has that would disqualify the use of routing ASICs.

We obviously have very different perspectives on this issue. I think given Google's architecture, where a host handles millions of flows and they are all individually keyed, just the key management nightmare alone would be enough to discredit the idea of offloading crypto to an ASIC in a separate box.

Does that actually accomplish the desired goal? Encrypting across data centers is one thing, but encrypting within a data center only makes sense if the threat model includes the datacenter's internal network being compromised – I suppose either physically or by hacking a router or something. I can't judge how much that threat model makes sense. But to the extent it does, a centralized router doesn't protect against it; in fact, if you're going to trust the router(s), there's no point encrypting the data at all. On the other hand, a per-server smart NIC would achieve equivalent protection, but maybe that's more expensive?

If you split secrets across NICs such that compromise of one NIC would only compromise a portion of internal datacenter traffic, then that would make sense. Again, as you noted, it depends on the threat model.

But from just the bandwidth and latency perspective, a custom ASIC makes more sense to me.

Theyre google, they are totally down with a 1% trade for flexible and maintainable systems.

but getting from 0..custom-asic is waay more harder than you make it out :o) plus you have a _custom_ stack on top of it, which is not malleable at all. moreover, getting a vendor lockin is probably not such a good idea anyways.

giving credit where it is due, intel's role in development of latest pcie-4.0 and upcoming pcie-5.0 specs and ofcourse the latest cpu's make it less and less appealing to consider putting up with custom h/w + s/w imho

with dpdk for example, doing 10g/core with 64b packet sizes is considered pretty standard these days. and if you don't want to roll your own, you always have companies like 6wind to help you (and ofcourse they share the code as well)

facebook/google etc. have, by now, probably at least a decade of experience running their own sdn enabled networks. facebook even has fboss to commoditize switch from white-label manufacturers...

SONiC is an interesting initiative by MS that reduces the impact of vendor lock-in by abstracting out ASIC specifics: https://azure.github.io/SONiC/

Even if you utilize every last cycle of CPU, the price/bps and energy/bit will still be way higher than almost any switching or routing ASIC on the market.

That doesn't mean you're doing crypto with integer operations. Newer CPUs support AES in hardware.

Well, of course. Practically all of the primitives in openssl boringssl etc are coded in assembly.

> This sounds like it’s doing packet processing in software

From the article:

> Based on a custom implementation of IPsec

Seems pretty clear.

>custom implementation of IPsec

What could possibly go wrong.

I'm sure this is just referring to various 'extension headers' in IPSec, which is how the protocol was designed.

Sure, poor implementations can lead to problems, but the same holds true for just about anything.

Reply on news.ycombinator.com