Hacker News Reader

Comment by andrewstuart2 | Hacker News Reader

In addition to being so small (text-based, minimal JS), HN is cached by CloudFlare, which grabs copies of dynamic pages and pushes those copies out to their CDN edge servers around the globe so they're closer to end users.

When your browser asks for the IP to news.ycombinator.com, you get the address of a cloudflare CDN server. Which web server the DNS server will send you is based on a GeoIP lookup of your own IP address so that they can give you the address to the server physically closest to you, and therefore the lowest possible latency.

Edit: Apparently this is not how CF does things (see below explanation by benjojo12) but is not uncommon in load-balancing and distribution.

    news.ycombinator.com.	296	IN	CNAME	news.ycombinator.com.cdn.cloudflare.net.
    news.ycombinator.com.cdn.cloudflare.net. 299 IN	A 198.41.190.47
    news.ycombinator.com.cdn.cloudflare.net. 299 IN	A 198.41.191.47

View on HN

Pretty close!

We don't actually use GeoIP at the DNS level, we use the routing table of the internet to direct people to the closest location we have to them by announcing the same IP addresses in the routing table in many places (eg, anycast) You can find a blog post primer and general more explanation here: https://blog.cloudflare.com/cloudflares-architecture-elimina...

the upshot of this is that TCP and SSL connections start faster because of lower latency to our location, and we can pull items out of cache faster and give them to the client.

Awesome, thanks for the additional info. On a related note, I really appreciate how good CloudFlare is about knowledge sharing. Not a lot of companies would be so willing to share information that's so integral to their business.

although it's useful to note that use of anycast is impossible to hide. any looking glass will tell you that CF has tons of routes to each apparent IP, e.g. https://ssp.pme.gin.ntt.net/lg/lg.cgi?query=bgp&sourceIP=IP&...

On the other hand, they do some stupid stuff, disregarding the standard and how useful it is [0].

[0] https://blog.cloudflare.com/deprecating-dns-any-meta-query-t...

Disregarding standards is not uncommon when you're actually running a high-performance system that faces attacks. Plenty of stuff goes into specs that just doesn't make sense while operating. Their explanation of why they are disabling ANY seems perfectly fine.

All I can see is that you've never needed to debug problems with DNS.

So every time a new comment is posted, is the CloudFlare cache invalidated and replaced with a new version of the comments page?

Actually for now, HN is not caching their HTML with us. we put a "CF-Cache-Status" header for when we are caching things

So when hitting a comment page, HN's backend is being hit in some way, whether by reading a db or by concatenating flat files or something else? Or are they just using a local cache that isn't pushed out to CloudFlare because there would be so much churn that pushing to CF wouldn't be a benefit?

Are they using Railgun to only transmit changes? Railgun seems like the neatest thing CF offers, as, AFAIK, you get the bandwidth benefits of caching without having to deal with invalidation.

Not sure, but cloudfare supposedly speeds up content delivery even for dynamic content where you need a new db hit each time. The idea is that it leaves open a persistent connection with the webserver, which means A) less work creating and destroying connections B) each new connection isn't starting off as throttled.

They have tech to do even better. See here https://www.cloudflare.com/railgun

That is how you can make it faster even when somebody is logged in and you are delivering a somewhat personalized experience.

Reply on news.ycombinator.com