Hacker News Reader

Comment by 616c | Hacker News Reader

Interesting, I was expecting to see something about QUIC/UDP and was pleasantly disturbed by something different entirely.

First takeaway: SSL is optional? Let's the games begin! Better pay your ISPs on time, or God knows what they will accuse you of, extralegally or not, to get encryption turned off here.

Second takeaway: our company had major problems with upstream link from a local ISP to VPN across the Atlantic to the link on the other end and egress traffic out of our main office. It is really more than one, but you get the idea. The ISP in the US mentioned "caching problems" when we had many users with irritably high latency for Google servers: slow page loads, terribly syncing, crappy IMAP, but only for Google services we use. This data compression stuff seems limited. Does Google have boxes on-prem for ISPs a la Netflix that do the same thing or is improperly configuring this for an ISP enough to do damage?

https://openconnect.netflix.com/

View on HN

Yes its called GGC. https://peering.google.com/#/

We got ours delivered about 6 months ago. It usually works fine but once in a while google services will act up, but most of the time the issue is at ISP end because even the GGC nodes needs BW to pull non-cached content, but if your ISP is overselling BW it will really hamper the performance on the GGC nodes. The GGC nodes themselves have capacity limitation, our 3 nodes has a total capacity of 19gbps, I am guessing if we were to saturate that throughput then the services will be affected.

That's interesting, mind expanding on your problems?

I know that as a BT (uk) ISP customer mostly I can ping google.com and it returns in 6ms from a BT IP, which would indicate they're using GGC. Sometimes it resolves to a Google IP in 12 ms.

Last month however it started resolving all the way to Amsterdam and subsequently to the US with a corresponding increase in ping time. Was this their way of saving energy by shutting down DCs, or was it a bigger problem like capacity or an outage? Either way despite the latency issues the fact it always works in a plus for their infrastructure team.

If you are using opendns you should be hitting gcc usually.

I am sorry I cant get in to specifics. But Routing for GGC is controlled by Google, it is not out of ordinary for your google traffic to not come directly from the GGC node of you ISP. Any internal routing change by your ISP might also cause your Google traffic not coming from your ISP GGC node.

There are too many variables its hard to tell what exactly caused your issues.

But generally speaking GGC nodes are very reliable, their support is responsive and knows what they are doing very well and most importantly it saves us insane amount of money (cause BW is expensive here), and our users are getting better service.

Are you able to disclose if it caches/proxies YouTube?

Yes of course, the main traffic is youtube bw, which I think its 95% of the total traffic going through ggc nodes. But the don't have a breakdown of their traffic on GGC dashboard. We know it because we know our traffic before we had GGC and after we had GGC.

You might still get content from closer to home, because YouTube might redirect the heavy HTTP request back to an edge node. (Check the source of the actual many megabyte response.)

(Tedious disclaimer: my opinion only, not speaking for anybody else. I'm an SRE at Google, and I'm oncall for this service.)

The reasons why this happens are complex and difficult to debug. Please contact your ISP and ask them to pass the details of the problem on to us.

This seems to be an improvement. You trust Google's network, and you get a HTTP/2 proxy. (Or if 443 is blocked, it falls back to HTTP/1.1.) Much better than browsing plain HTTP on a shared (corporate or not) wifi.

Reply on news.ycombinator.com