Hacker News Reader

Comment by GauntletWizard | Hacker News Reader

One of the biggest things I've had to unlearn as an SRE leaving google is this: RPC traffic is not free, fast, and reliable (So long as you don't go cross-datacenter). For most companies it is expensive and slow. Facebook's networks are still designed for early-2000s era topologies and their newer topologies won't fix that; They've still got way too little top-of-rack bandwidth to the other racks nearby.

Microsoft hasn't even caught on yet, and is still designing for bigger and bigger monolithic servers. I can't tell what Amazon is doing, but they seem to have the idea with ELBs at multiple layers.

View on HN

This is what continues to shock me about the tech industry. Google started revolutionizing the way data centers work more than 15 years ago, and it's a huge reason why they are able to execute on ambitious projects so well and serve up content that has a ton of compute behind it yet do so incredibly fast yet also incredibly cheaply. That cuts right to the bottom line, yet nobody else has copied google yet. I think it's a huge indication of just how immature the entire industry truly is. It's difficult enough to get anything done these days, let alone get something exceptional done well. We're only just a little bit out of the deep depths of the "software crisis" where the ability to run a successful software project was a hit or miss thing. But we're still in the software crisis shallows, and it'll be a long time before we make land fall.

> nobody else has copied google yet

Why would they ?

Almost nobody is operating at Google's scale and interconnects like Infiniband are rarely saturated to the point that exotic solutions are needed. Fast RPC is nice and all but (a) not that many companies are using microservices and (b) not that many are scaling out to the billions of requests/second.

And the 20 or so companies who are operating at Google's scale are hardly floundering around blind. Facebook has done very interesting work in the infrastructure space, LinkedIn is amazing on the core software side and Netflix has pioneered the use of the cloud.

I think this is short-sighted. Everyone who operates with any portion of their stack on any cloud provider is "operating at Google's scale" since even a small service is affected both by the infrastructure on which it is run and all of the other services run on that infrastructure. If you're running in a cloud, you need to plan and develop like you're a product team at facebook or google, and cloud providers need to act like they are providing infrastructure for a cohesive system, not a bunch of individual pieces.

It's not short sighted. It's called not prematurely optimising.

There is a world of difference between the issues Facebook, Google, LinkedIn etc are dealing with and the rest of us. Nobody is worrying whether 10% of their CPU/Memory/Network capacity is going unused if we only have a handful of servers. Or overly worrying about interconnect bandwidth beyond a certain point.

It is fascinating and useful to see how to scale but we aren't all in the dark ages simply because we don't follow their architectural approaches.

No one is saying you should build this in your back office. My comment was limited (by the first bit) to people utilizing public clouds and those who run public clouds. If you are, you are affected by these issues; if you are not, then there's an opportunity cost there, whether or not you believe the early optimization is greater or lesser.

The two examples I think that matter are Amazon and Microsoft. I don't think they copied Google- I think they independently realized, when they decided to enter the cloud service, that they had to scale up and out their data centers tremendously. And when you do that, you quickly learn that deploying more hosts is easier than deploying the ports to interconnect those hosts with a globally nonblocking fabric.

Amazon and MSFT have both published nice papers showing that they "got it" about 3-5 years ago. I think they operate within an order of magnitude of Google in terms of total external and internal traffic, but that's just a guess because none of these companies publish enough detail to say for sure.

As a googler, I would agree with this bucketing. Amazon and Microsoft are in the same ballpark scale-wise, but companies like LinkedIn and Netflix have lower scaling needs (this isn't a knock on them, just a technical reality). Netflix would likely be in the same ballpark if they didn't leverage Amazon. Facebook is also hovering near the line, and has probably crossed over.

> RPC traffic is expensive and slow, not free, fast, reliable

If i've understood your point, you should have rather said this is what you've learned, not unlearned.

You are correct; I've edited my comment to (hopefully) make it a little clearer. Google RPCs are fast. The RPC trace depth of many requests is >20 in miliseconds. Google RPCs are free - Nobody budgets for intradatacenter traffic. Google RPCs are reliable - Most teams hold themselves to a SLA of 4 9s, as measured by their customers, and many see >5 as the usual state of affairs.

Is it opposite day? You seem to mean <20ms right?

I parsed his comment as "a request that results in 20 nested RPCs completes in a few ms".

I think he means that over 20 remote procedure calls can be made within milliseconds, which is game-changing.

Try this link for what MS is doing :

https://www.youtube.com/watch?v=RffHFIhg5Sc

It would be nice if the SREs (and SWEs) still at Google would also unlearn these incorrect facts. RPCs fail, bandwidth costs money.

Reply on news.ycombinator.com