Linode-hosted DNS zones were down | Hacker News Reader

Linode-hosted DNS zones were down

status.linode.com

43 pointsmwpmaybe48 comments

I can confirm this was not a DDoS, and we are actively working to fix the issue right now. We'll be updating our status page as soon as we have more updates to deliver.

Do you have any ETA? Sorry, I don't mean to be an ass, it's just our entire business is currently down because of this.

I understand, and I don't have an ETA quite yet, but it appears that service is already starting to return. I've tested my domains hosted on Linode and a number of others that I know are and they appear to be working now. We're still working asap to ensure everything is coming back appropriately.

Thank you for even that partial info. We can confirm that we're seeing some response now.

Everything seems to be back now in our case. Thanks!

Isn't half the point of having 5 bloody DNS servers is that they shouldn't _all_ go down at the same time?

Get it together Linode!

The internet facing part of Linode's DNS service is hosted by Cloudflare.

I don't understand... can you expand?

They have a pretty silly DNS reverse proxy setup with cloudflare.

Silly because their backend servers appear to still be on the internet, e.g. ns1.linode.com at 69.93.127.10

My monitors started going berserk around 3:36 PM CT. Here's an example of the problem (I am not hostmaster for The Onion, but they are a well-known Linode customer):

    $ whois theonion.com | grep -i 'name server'
       Name Server: NS1.LINODE.COM
       Name Server: NS2.LINODE.COM
       Name Server: NS3.LINODE.COM
       Name Server: NS4.LINODE.COM
       Name Server: NS5.LINODE.COM
    $ dig @ns1.linode.com theonion.com ns +short
    $ dig @ns1.linode.com theonion.com a +short

I prefer DJBDNS tools:

    #dnsq a theonion.com NS1.LINODE.COM
     1 theonion.com:
     temporary failure

This is one reason why I'm moving our startup over to AWS. :/ They seem to experience less of these catastrophic DDoS attacks.

Edit: Err, not a DDoS attack apparently. But catastrophic nonetheless.

Route53 also:

A) Serves DNS from multiple domain names. A number of variants in .com .org and others etc

B) Serves DNS from multiple internet networks / sources. Queries from an end user don't all hit the same Anycast subnet.

Route 53 also has no IPv6 support, and no stated policy on how it deals with DDoS attacks, or if you get charged for the attack if it's directed at you. Good luck!

It might be more accurate to say R53 has partial v6 support. It'll return v6 records (AAAA) happily. The nameservers themselves do not respond on v6 (which is unfortunate and certainly means v6 support is lacking). Practically, however, it's pretty unlikely, that a v6 only host isn't going to also have access to a v6-to-v4 gateway. Not ideal, but generally transparent for most intents and purposes ...and generally used by just about anyone with a v6 address assignment today, everyday.

For DDoS attacks, I'm not sure what you mean by a "stated policy." They do a lot of countermeasures, some of which can be found via linked PDFs at https://aws.amazon.com/security/ Various R53 items are included in their DDoS document there. Officially you get charged. Unofficially, it depends on the circumstances. There's also a world of difference between a DDoS attack against a fully managed offering (like R53) versus, say, an EC2 instance.

Have you seen a better approach taken by other providers? I don't mean technical approach (which is near impossible to compare other than by track record), but from a policy perspective?

At one point, this said it was due to a Denial of Service attack. That text has since been pulled, unfortunately I didn't get a screenshot.

The email I received:

DNS Performance Issue Incident Report for Linode Investigating We are currently experiencing denial of service attacks that are targeting our DNS infrastructure. Aug 23, 20:48 UTC This incident affects: Hosted DNS Service.

I recently migrated away from using Linode as secondary when they pushed everyone over to Cloudflare. That Internet-destroying company already has enough power!

Why do you think Cloudflare is internet-destroying? Serious question.

They are growing into a very dangerous position of acting as a gatekeeper to the Internet. Try accessing any major website through Tor.

And try having a webserver being constantly attacked by TOR.

They are not doing it for fun. If you think they are bad, give them suggestions or new tech how to solve this problem.

I thank cloudflare for making my life easier.

Not only a gatekeeper, but also a keymaster. (As it were.) Cloudflare's role as a reverse proxy, and as a SSL CA, and as a DNS provider leaves users and webmasters both highly vulnerable to surveillance and/or content tampering by CF.

Not saying that they're doing anything malicious, of course. Merely that it'd be very easy for them to do so.

So far Cloudflare appears to be acting in a non-destroying way. They take as content-neutral a stance as they can, and have had to defend lawsuits as a result of that stance. They recognize that the internet only functions and they can only be part of the infrastructure of the internet if they take that stance.

The greater concern is this: Companies like Cloudflare should not be necessary. They often are. And, with today's internet architecture, companies like Cloudflare must operate at scale to be a useful service with pricing that people can afford.

In a world where we accept the Cloudflare model, we also risk a set of gatekeepers emerging, and that set is "People who can manage terabits/second of inter-network connectivity at PoPs all over the world". It's potentially a troublingly small small group of entities.

Their service strongly undermines the end-to-end principle upon which the Internet is based.

It is the fragility of the TCP/IP infrastructure relative to DDOS attacks that is the problem.

I would love to hear about DDOS resilient service architectures that don't require something like Cloudflare.

A pre-funding project I am working will achieve DDoS resilience without gatekeepers like Cloudflare, if I am able to bring it to market.

It turns out that there's actually a ton of academic research on a number of very promising techniques, in addition to the approaches I'm hoping to develop.

I see the issue more as UI/UX/ease of use/cost of adoption. Right now a lot of approaches you could probably defend a Ph.D. on the topic. They need to instead become approaches developers choose because it's the easiest and most cost-effective way to deliver a high-quality service.

How does it work?

There's a lot of different technologies and ideas that need to come together in an easy to use way.

Three of the most important: * Blockchains * Protocol-level anti-DDoS changes (protocols specifically designed to make them less suitable for use in in DoS) * https://en.wikipedia.org/wiki/Content_centric_networking

I think content centric networking is very interesting, but my original comment was in the context of TCP/IP as it exists today.

My gut says that real solutions to DDOS problems are going to have to come from new approaches to networking in general and not just tweaks to TCP/IP or throwing massive amounts of infrastructure at the problem.

I think we fundamentally agree.

But I would point out a subtle distinction: I believe TCP/IP and DNS provide an adequate building block for what needs to happen in a backwards-compatible progressive enhancement way.

I distinguish this from IPv6 which - because of how we chose to implement it - has created a ton of chicken-egg issues with the incentives for adoption.

Bittorrent, Freenet, IPFS.

I know there are a lot of sheep to be shorn in the centralized httpasture, but that shouldn't prevent us earnestly appraising how various technologies effect society.

I don't think Bittorrent and IPFS really qualify for decentralisation of random content on public internet. Do you want to advertise to the world that you're distributing the content you found on a random service you visited? You can become responsible for redistribution of copyrighted just because the website includes it.

It could work of course for very restricted applications, like software updates, but general, unfiltered web? No thank you.

Unless you're thinking of lots of content mirroring services where you can publish your mirrored content... but then, what's the difference from cloudflare?

Nothing in the architecture of those protocols requires everyone to republish, especially publicly.

The difference is in the namespace of those protocols. Rather than just reusing L4 addresses, they create addresses that are conducive to decentralization.

The problem with site mirroring services is once again fundamentally comes down to the naming. Either the site operator has to setup and incorporate the mirrors, or site visitors have to discover and visit each mirror manually. There is no middleground where you can ad-hoc casually ask a peer for a copy and have the exact same functionality.

If you skip the P2P aspect of those networks, I'm not sure how much it really differs from distributing HTTP traffic over multiple domain names with possible anycast on IPs. You even get an automatic DNS split per region. What's the reason to go with IPFS / BT for distribution from only the endpoints that you control on the internet? (I do see good reasons for it in private networks)

Yes, it is non-trivial to setup, but if you need it, then automating this part may be more trivial than getting clients to switch to a new protocol.

The pathological case is the same (fill up publisher's pipes -> no publishing), but by giving clients flexibility on how to retrieve, they can set their own policy for how to augment this.

Some possible general alternatives:

- Wait to retrieve from publisher directly, never from third party cloud services

- Ask friends, if this conforms to their privacy goals. Perhaps via store-and-forward mix.

- Ask public altruistic peers.

- Ask for-profit "cache providers".

- Retrieve from multiple independent cloud service(s) using a technique such as PIR

As a strong believer in the end-to-end principle, I don't understand what you mean.

They have:

* A reverse proxy service, open to anyone

* A network-level traffic volume management system, open to those that can afford it

I can come up with a ton of scenarios where both of these services actually help with the whole 'end-to-end' issue.

It doesn't really matter who they'll provide service to, it matters how they alter the architecture of the network.

The two ends in a communication are the user and the site publisher [0]. Cloudflare inserts an intermediate proxy. It doesn't matter that it's woven in using IP, just how it doesn't matter that CableCo's proprietary VoD service delivers over IP.

[0] Or even two users wishing to chat or whatever - the same applies to the whole "web 2.0" bubble, but slightly less egregious than Cloudflare.

But do they actually alter the architecture of the internet?

Or rather, how is Cloudflare making things worse or substantially different?

Example plausible flows in the life of a packet or HTTP transaction:

Before: Packet goes Comcast->Level3->HE->Destination After: Packet goes Comcast->Cloudflare->HE->Destination

Note that Destination is the one choosing to replace Level3 with Cloudflare. If Cloudflare ceases to be a good choice, Destination can swap Cloudflare for an alternate provider.

The use case where Cloudflare handles the HTTPS decryption (MITM-as-a-service) is a little bit different. Although it's become quite common, that's not the only way Cloudflare can be used. And I agree that's something where it changes things for the worse in some cases (but if it's a case of "Amazon's machines decrypt it for you" vs "Cloudflare's machines decrypt it for you", I'm less clear that a new problem is being introduced by Cloudflare).

It's not a simple rerouting of a packet. It's a complete L5 decode/proxy/recode, inherently supporting only a few protocols. Putting this kind of functionality into the network is exactly what the end to end principle was reacting against.

(1) Cloudflare does do BGP-based protection (rerouting of a packet) on their enterprise tier.

(2) They really aren't adding functionality into the network. If I put a cluster of Nginx servers in front of my web server, or if I have Cloudflare do it for me, I don't see how that fundamentally changes anything.

(3) The Cloudflare approach of an application layer proxy is feasible for any protocol which can be proxied. I've worked on SMTP proxies which do to SMTP what Cloudflare does to HTTP. Cloudflare also has a DNS proxy. Proxying is a perfectly valid and often incredibly useful technique.

1. Sure, they can do multiple things. When most people talk about Cloudflare, they're referring to the L5 interdiction.

2. You could make a similar argument about network gear having code to facilitate streams. Having a third party provide this functionality is further down the path of centralization.

3. Sure. The point is that to create/tweak protocols, you have to convince Cloudflare (etc) to implement.

It's the end to end principle for a reason. It's not a hard and fast rule, in fact there are many immediate reasons for wanting centralization. It's just unsustainable in the far term.

Is not it possible to outsource that critical part? What are good and proofed dns services out there?

They do outsource to Cloudflare.

One assumes some behind the scenes infrastructure exists to push and sync updates to Cloudflare.

They probably use CloudFlare's DNS proxy service: https://www.cloudflare.com/virtual-dns/

Unless you want to rely on one company with anycast (like amazon or cloudflare), you can always just keep the same zone in a number of unrelated companies.

This was just the kick in the pants I needed to set up secondary DNS at Route 53!

Seems like it's coming back up! My resources are reachable again.

This smells like a major fuckup

Reply on news.ycombinator.com