Cloudflare was down | Hacker News Reader

Cloudflare was down

cloudflarestatus.com

926 pointsdewey476 comments

Discord is entirely down right now, both the website and the app itself. Amusingly, a lot of the sites that normally track outages are also down, which made me think it was my internet at first. Downdetector, monitortheinternet, etc.

Lots of other big sites that are down: Patreon, npmjs, DigitalOcean, Coinbase, Zendesk, Medium, GitLab (502), Fiverr, Upwork, Udemy

Edit: 15 min later, looks like things are starting to come back up

Hacker News is an excellent status page for those cases.

Out of curiosity, done HN use any CDN or other way of DDOS protection? dang?

My iPhone actually popped up a message saying that my wifi didn't appear to have internet, which was strange and obviously false as I was actively using the internet on it and the laptop next to it, but now it makes sense that it must have been pinging something backed by cloudflare!

Discord attempted to route me to: everydayconsumers.com/displaydirect2?tp1=b49ed5eb-cc44-427d-8d30-b279c92b00bb&kw=attorney&tg1=12570&tg2=216899.marlborotech.com_47.36.66.228&tg3=fbK3As-awso

(Visit at your own risk.)

Hack?

I'd be looking at your browser extensions or malware (if you use the Discord app).

Sure you didn't misspell discord?

How can this be reproduced?

Same here!

I even checked to see if an AWS region was down once I realised it wasn't on my side (I thought it might have been my ISP's DNS servers or something).

The next move was to check Hacker News - thankfully it's not also hosted on Cloudflare, ha!

I noticed discord being down, so I went to check downforeveryoneorjustme, also down. So I figured I'd check NANOG mailing list, also down :P

Yep, we were down completely. We are quite dependent on Cloudflare (frontend + dns).

https://status.discord.com/ is down. Wow.

And that is why you host your status page on separate infra.

It's hosted by statuspage.io, and their own status page was also down (metastatuspage.com). It is now back up, but their page shows the outage.

Works for me.

Works for me too... Australia

I can live without creepy instant messengers, but its shocking just how much everything else relies on one, central system. And furthermore, why is it always cloudflair?

Cloudflare is free and has a nice UI. I manage ~40 domains from ~six domain registrars through it, the consistency is great. The caching is a bonus.

Discord confirmed Cloudflare is also the reason they're down: https://twitter.com/discord/status/1284237737638461453

Doordash too. My first order. On my wife's birthday.

Ah, well. This too shall pass.

I did a pickup order a few weeks ago when of all things Tmobile SMS went down for 3+ hours. I couldn't go in the restaurant (covid) and I couldn't text them the parking # I was sitting at in a packed parking lot. I got a flood of about 50 texts a few hours later. Sat there for about an hour waiting for a $9 sandwich. I have no idea if they didn't get my order until late, or if they finally realized it was me or what. About 45 minutes in I decided to just give up on the day and take a nap, woke up to a door knock.

Kudos to the people at Discord. Just a few minutes after I got disconnected they already tweeted about the issue. Some minutes later and they have a message in their desktop app confirming it's an issue with Cloudflare. All while Cloudflare's statuspage says there are 'minor outages'.

Every company rushes to report an outage when they can blame another vendor, well that might be hyperbolic but it's sure a lot easier!

As a percentage of total traffic, a 'minor' outage for Cloudflare probably equates to a significant outage for a non-trivial amount of the internet.

It will also be especially noticeable to end-users, because sites using Cloudflare are typically high-traffic sites, and so a 'minor' issue that affects only a handful of sites is still going to be noticed by millions of people.

I wonder if they are all using Cloudflare's free DNS stuff or if they're paying for business accounts?

My stuff is on Netlify (for the next week or so) and the rest is on a VPS bought from a local business who isn't reselling cloud resources. I'm kinda glad I moved all my stuff from cloudflare.

I think it's going to be everyone. Some of my free sites are dead, but also huge enterprise Cloudflare users (Discord/Patreon/4chan) are also dead.

> Amusingly, a lot of the sites that normally track outages are also down, which made me think it was my internet at first.

That is why if you have this question, you should go to google.com

My guess is that there are more resources invested in making sure google.com stays up than for any other site on the internet.

Depending on what part we're talking about, it varies. But yeah, just a few.

Crazily, my local name resolution started failing, because I have these names servers: 192.168.0.99, 1.1.1.1 and 8.8.8.8. The first does the local resolution, but macOS wasn't consulting it because 1.1.1.1 was failing?? Crazy. When I removed 1.1.1.1 from the list, everything started working.

DNS over HTTP might bypass your local nameservers.

Thought something like this was going on. At first I thought it was my router and restarted everything - to no avail. Glad to see confirmation that it wasn't an issue on my end.

Discord works for me, but https://redbubble.com/ prints "Service unavailable".

Freenode's IRC servers were down which was unexpected for me. I was expecting old-school communication networks to not have a dependency on Cloudflare.

I've had no connection interruptions to the three IRC networks I'm connected to. Freenode, EFnet and Hackint.

I loathe Discord, and I can barely contain myself with schadenfreude at this news.

Ironically, downdetector.com is also down.

Who watches the watchmen?

Would IRC be down?

Same for the German downtime trackers.

same here. I tried if hacker news still works and saw this

It really defies the original vision of the internet to have so many services depend on a single company. Almost every news site I was reading dropped off at once. I thought for a second that I lost internet in my own house.

Yes its really odd that core backbone providers can go down and everything works like its supposed to. Even trans-pacific cables can be cut and things will usually work with only increased latency. But there is not much redundancy for many companies at this layer; having redundant DNS providers is I'm sure possible but not something we think about very often, and of course many of the sites that are down are depending on the proxy and DOS mitigation services.

On my home network I use Google as a backup DNS provider so the whole internet didn't go dark for me, but I don't have a backup DNS host for my company's DNS records.

Agreed, but the real problem is DDoS and nobody seems to know how to globally solve it. Fighting DDoS is expensive, so you see consolidation. It's well and good to live in a tiny farming town but when raiders start attacking every week, those castle walls and guards start to look really appealing.

Agreed, but I think people really underestimated the forces at work that would cause so much consolidation into a couple internet giants.

The original idea was that with the barrier to entry being so low, anyone and everyone could set up their own websites, mail servers, etc.

But with it being so easy to compare and contrast service (i.e. the market being so open), it means that the competitive forces naturally consolidate to a winner-take-all model. If when starting out Cloudflare was just 5% better than the competition, it could have easily taken the vast majority of the mindshare on the internet. Couple that with the fact that there are huge advantages with scale to a business like Cloudflare's, and it's not hard to see how so much of the internet has become dependent on it.

Same here. Rebooted the router and modem thinking it was me, but my phone was still on wifi then realized it was probably my cloudflare DNS.

I consider DNS and the way how top level domains are handled to be one of the weakest parts of our current Internet design.

We REALLY need a truly decentralized, distributed DNS system that is not owned by private entities.

And the worst is if you try to raise concerns about cloudflare now it get brushed of as "cf already proxy half the internet, if it goes down our stuff will be minor concern".

I don't understand why the big companies don't always have at least two CDN providers, so they can failover to another one if something like this happens.

I know a lot of big companies do, but I am always surprised when you see ones that don't.

My CRM was nonfunctional. That’s some critical infrastructure for me. And then I’m wondering, is it me or is it my CRM. Turns out it’s door #3 - cloudflare

Same here. I'm working at an auto parts store looking though ASE parts sites and it was like well close up the store the catalogs are missing RN.

"All systems operational"

What's the point of a status page if it doesn't reflect the real status...

It's either the status page goes down with everything else or the status page is wrong. Great.

EDIT: Looks like it's accurate now, 20 minutes later.

The point of the status page is so you can point to it for your five nines SLA and go "look? we were only down for one hour". As soon as the money relies on the metric, the metric will reflect the money.

Goodhart's Law[1] in action.

[1]https://en.wikipedia.org/wiki/Goodhart%27s_law

Despite their update, I like how they're saying only their recursive DNS had "degraded performance", while authoritative is "operational". The entire reason everything blew up was because their authoritative nameservers weren't responding.

IBM Cloud status is pretty much always green... although we have issues pretty much every week.

They’re still using Lotus Notes for the tracking.

There are status page providers that actually monitor services and automatically update. Cloudfare just doesn't use them.

Let's start a betting pool. How many upvotes do you think OP will get before the status page acknowledges a problem? I say its going to be 600.

You lost. ;) 476 points, status page says it's down now.

This post is getting something like 30 upvotes a minute…might want to up that a bit ;)

And it looks like they started "investigating" at around 450!

Ahh I remember when AWS went down (think it was 2 years ago now?) or at least a data center in us-east? Majority of the internet went down and status page went down as well. Man good times.

Status pages are a marketing channel not a channel for developers most of the time. It most likely has to go through some layers before someone updates the status page.

This is an Atlassian Statuspage status page, so it's not hosted by Cloudflare.

this-is-fine.gif

I don't think it's just Cloudflare; I just had a fun 10 minutes seeing servers start flipping on my Server monitoring service[1]. This has only happened once or twice per year, and is usually due to weird global DNS issues.

[1] https://servercheck.in/

(To give an update, I'm seeing from my monitoring systems (about 15 points around the globe) sporadic outages for Microsoft, Apple, Reddit, Bing, Node.js, Twitter, Yahoo, and YouTube. And my own servers (not behind CF at all) are also flipping up and down. It started around 21:14 UTC.)

a DNS issue wouldn't cripple all of the internet at once, with all the caching.

Most sites set the absolute minimum TTL for every record, for no reason. There’s a lot less caching than you’re thinking.

It was interesting that we saw our domains affected from the USA but from Mexico everything looked OK.

The crazier thing is that I tried to login to our CloudFlare account, it never sent me the 2FA code... I still haven't been able to login (Enterprise account)

This was a problem with our backbone network; wasn't caused by an attack. The effect was regional and not global. Naturally, we'll write it all up.

Was it a problem with a provider you use?

Looks like problem with one of our large routers in Atlanta.

We were down (downforeveryoneorjustme.com) completely, but back up now (as of a few minutes ago). Our domain wasn't even resolving; we use Cloudflare for frontend and DNS.

We had a surge of people checking if Discord was down on our site, then I noticed everything went down shortly after. Discord is still the top check right now.

I can't ever remember hitting these kind of traffic numbers before.

Interesting data you get in the face of adversity, providing your host resolves!

Funny, I tried to use your site because the website I was trying to access stopped working. But your site was also down so I figured it was just my internet being cranky :/

I enjoy your service. Have you ever thought about expanding your offerings? I would love to see a recreation of "Internet Pulse"

Something’s wonky, because it’s not just Cloudflare. One of my personal sites is down that uses nothing but a VPS, and I noticed my Unifi AP disconnect from its controller a little bit ago. Fiber cut? Routing issues?

If that VPS is on DO they're down too cause of CF. Or if you set the resolver on your VPS to 1.1.1.1 that's also down.

Why are digital ocean VPSes down due to a Cloudflare outage? Hoping for a clarifying post mortem...

DO is still up as my machines are still up and accessible.

Huh. My Ubiquiti was reporting WAN link down during this outage. I'm using ATT fiber. I'm wondering if "link down" doesn't mean what I think it means. Now that I check, it says "WAN iface [eth2] transition to state [inactive]". I'm wondering if that means link down or if it's doing service checking.

I actually have a WAN2 configured but not plugged in and it was set to "Load Balancing: Failover Only" ... I wonder if all of my 'connection issues' were software assuming my network link is down and switching interfaces to an unavailable one.

to reply to myself, if you have a second interface configured for failover, it actually tests against ping.ubnt.com. I bet every single time my ATT fiber has "gone out" for a minute or two at a time, it's been bogus.

  root@USG-PRO-4:~# show load-balance watchdog
  Group wan_failover
  eth2
  status: Running
  pings: 2
  fails: 0
  run fails: 0/3
  route drops: 0
  ping gateway: ping.ubnt.com - REACHABLE

  eth3
  status: Waiting on recovery (0/3)
  failover-only mode
  pings: 1
  fails: 1
  run fails: 3/3
  route drops: 1
  ping gateway: ping.ubnt.com - DOWN
  last route drop   : Fri Jul 17 17:32:58 2020

We can't keep going on like this. The vulnerability of centralised internet infrastructure is a huge problem for everyone. Somebody, somewhere, really ought to sort it all out

> Somebody, somewhere, really ought to sort it all out

That could be the slogan for 2020

10-20 minute router misconfigurations and subsequent fixes are sometimes a fact of life. big network infrastructure is complicated, and sometimes the best laid route tables of mice and men do go abloop and die.

Outages happen no matter what the infrastructure is. There's no solution, they're just something you need to recognize and handle, which Cloudflare seemingly did relatively quickly here.

If it impacts enough wallets, things might change. I'm not holding my breath though.

Why not you? Just don't use CF. The more people stay away from CF, the better.

Sounds like a problem for... us!

Be the change you want to see in the world :) There are no somebodys somewheres.

One question is how to do DDoS protection without somebody like Cloudflare. Some new protocol for edge caching, perhaps?

Or just stop using the internet. The majority of tech problems stem from people using tech. Don't rely on it alone and you don't have problems.

Yesterday I noticed most of their lava lamps are out (which generate random bits). Perhaps these are a critical component.

https://photos.app.goo.gl/g6eR8V2PSY3EVjCLA

I'm sure you were joking but they actually are: https://blog.cloudflare.com/lavarand-in-production-the-nitty...

Despite them actually mixing it into their entropy pools, the lava lamps are still entirely for show. The noise of the camera sensor itself is going to contribute orders of magnitude more entropy than the slow movement of the lamps. It's not completely a fake stunt, but it's certainly headline-optimized.

Someone needs to get these lava lamps plugged back in ASAP!

Good guy cloudflare giving programmers an early weekend.

But overtime for their own programmers :)

A tweet from Cloudflare's CEO:

> Appears that the router in Atlanta announced bad routes (effectively a route leak). Only impacted our backbone. Not all of our PoPs are connected to our backbone, so some would not have seen an issue. Appears to have impacted about 50% of our traffic for a bit over 20 min.

https://twitter.com/eastdakota/status/1284259895475236865

I am probably the only one who cares but even https://downforeveryoneorjustme.com/ is down

I personally find sites like https://outage.report or https://downdetector.com which tally up the number, regions and history of people saying it isn't working for them more conclusive.

Now we need isdownforeveryoneorjustmedown.com

Yep, we are very dependent on Cloudflare :(

It's up for me.

First place I went to also. Then HN.

Weird, it works for me (US)

That page still shows "Cloudflare System Status: All Systems Operational" for me, but it's definitely down for me. Along with 1.1.1.1, which is... bad.

For me it says "Minor System Outage" for about 0.1s and then shows "All Systems Operational".

Having DNS issues. Had to switch to Google's DNS (8.8.8.8/8.8.4.4) as 1.1.1.1/etc were not resolving anything.

Same. Even then, a bunch of sites are down. Maybe only ones behind Cloudflare? So far I've been trying to hit the various down detector sites and none of them will load. Google, Reddit, Hackernews are all fine.

Same. Had to go through the entire troubleshooting process --- is it my internet connection? my DNS resolver? firewall? ISP starting to filter DoT queries somehow?

Only last in my mental list was the possibility that Cloudflare would be down.

Hope they publish a detailed post-mortem. It's always fun to read (but certainly very painful for those directly involved in writing it).

1.1.1.1 is dropping ICMP pings while 8.8.8.8 is not but 8.8.8.8 is still returning DNS errors.

That only worked until all the records expired from Google's cache.

Monday Morning RCA: "We pushed out some routine code updates, but this really weird thing happened causing a resource utilization spike on our DNS systems. Because of this other really weird thing, this affected all of our global infrastructure simultaneously. Here's a deep engineering dive into this one weird thing that brought everything down."

Cloudflare's DNS (1.1.1.1) is failing to respond to most/all queries, which I'm observing as the root cause of a bunch of connection issues (name lookup failure).

Interestingly the same domains don't show up on google's (8.8.8.8) DNS at all.

8.8.8.8 is a caching resolver, it still needs to talk to CF's nameservers for authoritative records.

Lol, talk about timing. I'm currently working on a TLS library and was pulling my hair out trying to figure out why tests against CF sites suddenly failed. Can't even ask my cohorts on Discord because they are behind CF, too!

"StackOverflow devs have the most difficult job in the world. After all, when StackOverflow is down, they can't exactly look for help on StackOverflow".

No kidding! We had literally deployed a major page redesign and started watching our analytics drop off on it's way to zero. My heart is racing still. I wouldn't normally be happy for a cloudflare outage but in this case it's better than Google deciding to remove us from their index.

I can't change my NS records to point to a different DNS provider because my registrar, Namecheap, also uses Cloudflare. Didn't expect that.

Yep, Cloudflare, DigitalOcean and 1.1.1.1 down for me. I thought it was my internet and was so confused for a bit there.

Friendly reminder (and notes to myself):

Don't use Namecheap and Cloudflare at the same time.

Namecheap is using cloudflare. So if cloudflare is down, you can't change DNS settings on Namecheap as well!

Reminder for firefox users: Firefox uses DNS over HTTPS and the default is cloudflare. If you're having DNS issues, you need to disable it until cloudflare is back up.

Oopsie Daisy half the internet goes down

2020 is all in on everything bad that can happen.

Eggs and baskets etc etc

What luck, I chose today to install a new piece of network gear. I thought I had managed to totally FUBAR my network. DNS was failing, "ping 1.1" (my current goto test "Am I connected to the internet?" as it requires the fewest keystrokes and hits the Cloudflare DNS 1.0.0.1) failed and I just assumed it was my fault. Backed out my changes, and discover in fact, the internet was down.

Ping 1.1... thanks for that!

I was trying to fix my router the last 15 minutes :)

Yeah I was waiting for it to fix then tried cell phone and realized that was down too. I assumed it was an issue regional / backbone routing or something. Especially cause status pages which I wouldn't expect to be hosted on AWS (because of the need for status pages to stay up when AWS goes down) seemed to also be down. Didn't realize it could be Cloudflare...

Same here. Only figured it out because just one of the computers uses Cloudflare dns and the others were fine...

Our TPU management page is also down: https://www.tensorfork.com/tpus

Seems cloudflare took out a good chunk of the internet temporarily.

Doesn’t HN use cloudflare? Why did it survive? (I haven’t looked for about a year, but I seem to remember HN being proxied behind CF at one point.)

HN went off Cloudflare a couple years ago.

https://news.ycombinator.com/item?id=18188832

https://news.ycombinator.com/item?id=21799045

It doesn't look like it does. HN's IP space belongs to "M5 Computer Security", and their DNS nameservers are on "awsdns". Nothing there to suggest CF.

Is there a status page for HN?

HN allegedly still runs on one machine running a single-threaded Lisp webserver.

Parts of the site that are behind CF like the API are down.

Is it me, or has this been happening way too frequently for them lately?

To be fair the last major outage they had was 1-2 years ago. That said, when that happened they had two outages in about a month.

Honestly at their scale once a decade would be too frequent. Too many eggs in this particular basket.

Appears to be working for me now

; <<>> DiG 9.10.6 <<>> discordapp.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8092 ;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;discordapp.com. IN A

;; ANSWER SECTION: discordapp.com. 140 IN A 162.159.135.233 discordapp.com. 140 IN A 162.159.129.233 discordapp.com. 140 IN A 162.159.130.233 discordapp.com. 140 IN A 162.159.134.233 discordapp.com. 140 IN A 162.159.133.233

;; Query time: 69 msec ;; SERVER: 1.1.1.1#53(1.1.1.1) ;; WHEN: Fri Jul 17 14:37:40 PDT 2020 ;; MSG SIZE rcvd: 137

1.1.1.1 is not resolving anything for me at this time.

For people who reports this is down, which country are you in? Because all the reported sites works flawless from Norway (Europe) :-)

Brazil here, basically everything down.

I noticed a lot of packet loss to 1.1.1.1, not an outright "outage", maybe they were rolling a deployment?

Edit: Looks like a deployment to me (looking at the logs I could see cascading traces, so it took down one DC and the other started responding - increased latency - and then down, etc..), gonna be an interesting post-mortem!

East Coast USA here. Any cloudflare site is unreachable, and 1.1.1.1 is giving me massive latency and packet loss :)

US here, everything reported that I've checked is down. My Cloudflare sites are down as well.

I'm in Holland but everything came back up just now, so maybe you picked the exact right moment to check?

I put my bet on some peering fuckups, causing outrages since people are having packet loss etc.

Germany, everything down.

Ireland here, and it's all down.

Estonia, everything working fine

> * This afternoon we saw an outage across some parts of our network. It was not as a result of an attack. It appears a router on our global backbone announced bad routes and caused some portions of the network to not be available. We believe we have addressed the root cause and are monitoring systems for stability now.* Jul 17, 22:09 UTC

A lot of unusual internal traffic seems to be around Thailand (you might have to select "UNUSUAL"):

https://www.digitalattackmap.com/#anim=1&color=0&country=ALL...

been pretty large numbers from thailand this month according to that tool

Update - This afternoon we saw an outage across some parts of our network. It was not as a result of an attack. It appears a router on our global backbone announced bad routes and caused some potions of the network to not be available. We believe we have addressed the root cause and monitoring systems for stability now.

Their status page says that everything is operational. So much for a status page when half of the internet breaks down.

What I found particularly interesting is that my MacBook Pro (work laptop) didn't start up anymore properly and I wasn't able to start applications ... sorry, but wtf. Now I hate Apple and their shitty, overpriced products even more.

What was interesting and scary is that our monitoring system didn’t notify us. Our email was down because we use cloudflare for DNS and our monitoring provider’s SMS gateway was down. So we didn’t get sms messages.

Another useless status site.

DNS is completely broken.

"All systems operational" in nice soothing green.

No, not so much.

FYI, PagerDuty is not loading!

Time to go back to the drawing board, for a lot of us, to re-assess points of failure.

Edit: many websites are failing to DNS resolve but the services they provide continue to function fine behind the curtain.

This is likely your computer's DNS resolver (if you're using 1.1.1.1 you're down. I'd switch to 8.8.8.8 temporarily. We've had pagerduty alerts coming in since the start (a whole bunch of DNS errors from pingdom) and when I click on the slack link, pagerduty works for me

Interestingly this seemed to only affect resolver service. I use Cloudflare pretty extensively on all my sites, but only in DNS mode (no CDN / proxy). The hosts continued to resolve fine during the outage (following root DNS resolution chain, no recursive resolver involved). I imagine their CDN internally uses their resolver service which explains the outages, and some unrelated 3rd parties who don't use CF on their domain at all still created a hard dependency on CF by using their recursive DNS server.

Those poor Firefox users who enabled DoH

Happy Friday, everyone!

Ironically https://downforeveryoneorjustme.com/ is also down. HA

I'm having real problems with DNS, is this Cloudflare too? They say "All Systems Operational", so maybe not?

Half the damn internet is not currently resolving.

Yes, same here. Changed DNSs to level 3, all better now.

More like the Internet is down.

When you depend on a single company for much of the internet, such things happen :(

Site that use Discord, Linode, Patreon, npmjs, DigitalOcean, Coinbase, Zendesk, Medium, Gitlab (502), Fiverr, Upwork, Udemy and many more including 1.1.1.1 dns down. Ref: https://twitter.com/nixcraft/status/1284239374809395200?s=19

Seeing a lot of people mentioning DO, but it has been up for me without any issues (small VPS in SF-2)

My Pagerduty's been blowing up so I tried to go to their dashboard to pause the notifications for now and pagerduty.com is down XD

340ms average latency to 1.1.1.1 and 47% packet loss. Many sites are down. But I guess that's the problem with CDNs.

It's unfortunate that both the primary and secondary cloudflare DNS is down. I just switched my secondary to google.

This allows my internet to "work" during this time, but adds about 1s latency to resolutions. Presumably that's the time it takes my internal DNS resolver to try the secondary.

Considering running your own full resolver like unbound. Then you don't have to rely on a DNS provider like Google or Cloudflare. It's really nice not having the whole internet go down when Google or Cloudflare DNS is down.

DNS seems to be dead, if you have stuff in your cache and the site isn't low-TTL things still kinda work

Yeah this is absolutely killing us right now.

Looks like Digital Ocean is reporting an issue with their upstream provider:

https://status.digitalocean.com/incidents/6wtmldty17g1

As big as this is, any chance a major hub/backbone went down?

I don't use Cloudflare, but I do notice Cloudflare services being down.

Right now, I can't get to my own website (hosted on DigitalOcean, not through Cloudflare), but Oh Dear claims it's up. So I suspect that the problem is closer to me than it is to DigitalOcean (or Cloudflare).

DO uses cloudflare for their DNS... both for digitalocean.com and for their DNS service.

DO might use 1.1.1.1 (or Argon even) for routing between some of their PoPs

Aside from this one issue, is switching to 1.1.1.1 a good idea in your guys experience? Right now I just realized I hvae the DNS for my ISP which is probably how they inject bullshit 404 pages full of ads. What is the fastest/best public DNS in your guys experience?

I've been pretty happy with 1.1.1.1 (before now). Might be worth using something like 8.8.8.8 as a backup (Google).

I've been using https://nextdns.io/ - works fast and most importantly blocks a bunch of adds (user configurable), so makes browsing on mobile much nicer.

Ancedotally the interenet seems faster.

For us, it's cloudflare. Our ISP is connected to KCIX, and cloudflare apparently has 1.1.1.1 servers in Kansas City. No other free DNS provider is as quick, for us. 18ms or so RTT, as opposed to the WISP's internal latency of ~10ms. Central Kansas.

Looks like it's back. No longer getting issues with 1.1.1.1 and domains are being resolved!

Not for me, `dig @1.1.1.1 google.com` is returning SERVFAIL still. Their anycast config may be broken in some way (ie. the backends for some regions are down, but still advertising routes)

I thought my issue was with Comcast, then I realized I'm using CF's DNS entries for my home network. I removed those 1.1.1.1 entries and some sites are working.

Dang, I'm pretty disappointed in CF. I've never experienced this much an effectful DNS outage.

From what I can see externally this looks like DNS.

I wonder if that includes the roots that Cloudflare operate.

It’s worth mentioning here that 1.1.1.1 is also affected by this outage which initially made me think my internet was gone completely.

Changing back to an alternative (such as 8.8.8.8 from google) restored my access to the areas of the internet not using Cloudflare.

This was especially bad because I use Cloudflare public DNS exclusively at my house and it went down as well. I didn’t even think to check DNS, I just assumed it was AT&T being shiest AT&T.

I should probably run a blend of 1.1.1.1 with 8.8.8.8 instead.

Last I read about 7 million hosts are behind Cloudflare. Maybe around 3% of the web, but who knows if that counts for critical assets etc rather than pages served.

Shameful that so much of our decentralised web is so centralised and breakable in one place.

Just got a whole bunch of alerts that my services are down. Tried logging into Digital Ocean (who it seems uses Cloudflare) to get it fixed. Could not access their dashboard to reroute things.

RIP someone's weekend

Guess that explains Discord vanishing from the net a few minutes ago.

I'm surprised so many people still use them. They took my business down (along with half the internet) a few years ago and I learned that they were to large of a point of failure.

Ah Shit, Here We Go Again

Vercel also appears to be dropping out and coming back in intermittently over the last 30 minutes or so. Not aware they're using Cloudflare, although they do mention using AWS.

NextDNS got taken out by this, id been really happy with it up until now. And unfortunately “dns service went down” has a wide enough blast radius at home now that it’s a real pain.

How did you verify that? I determined the issue was with Cloudflare's DNS by toggling on NextDNS, which worked and continues to.

this is great, i already have bad enough internet (rural area with 3 to 6 digits latency and average 4 digits, barely a few kilobytes of speed) and having both google smearing everywhere their recaptchas that are not really friendly toward low speed internet / non chrome users and cloudflare proxying half the internet but lately not really doing a great job at keeping a consistent uptime does not help much

at least i am glad hn exists, it is the only thing that loads everywhere

I was trying to play video games but couldn't connect. Amazing how connected the web is now - one big hub goes down and brings the whole house of cards down with it.

Did anyone else see their ATT internet go down? The DNS issues started and then the Pace 5268AC rebooted. I don't use cloudflare for dns. Does ATT's backend?

On the contrary, ATT actually squats on the CloudFlare DNS IP address. IIRC that modem is one of the affected ones where it uses 1.1.1.0/24 internally. You shouldn't even be able to use CloudFlare DNS normally.

It shows "Minor system outage" when I load the page, but it switches to "All systems operational" immediately. Same behaviour on several attempts.

The status page linked shows "All Systems Operational" for me. Tested in private browsing and on my mobile.

Looks like DNS issues, their nameservers aren't reachable.

A lot of people are saying AWS. I'm having intermittent network connectivity issues intra-AZ, so perhaps they lost a data center or route flapped one.

What's all this about ”building a better internet”? Wide-reaching general service outages that are invisible to your status page are really not great.

I wonder if they'll cover why their status page is a steaming pile of garbage in the post-mortem?

Yesterday CloudFlare took down some of our products because they (not us) misconfigured some DNS thing. Kind of funny to see it happen again a day later.

This is hitting my production environments as well :-(

Twitch.tv channels are like 50/50 right now. Some are ok, some aren't.

Basically all Riot Games (League, Valorant, TFT) are down, dunno about LoR.

That would explain why Patreon is down. I was going to post a little frog I took a picture of on Lens. Went down just as I opened the app.

Hopefully with this outage Cloudflare will finally provide non-Enterprise plans a CNAME record, allowing us to quickly bypass Cloudflare.

Thought I was going crazy for a second.

This affects so many things it's scary, and Cloudflare status page has still not updated. HN got there first.

Interesting. They had an outage in the midst of a negotiation I was a part of. Are they less stable than Akamai and the others?

My modem also disconnected with signal problems, which was interesting. I'm not sure Cloudflare could have caused that?

maybe your modem uses 1.1.1.1 dns?

I was paying a bill in a bar in London about an hour ago and they couldn't process any payments. Seems likely related.

Must be regional or some other factor involved. Various sites others are reporting as offline load for me as does 1.1.1.1.

Oh, did hacker season just start? * Grabbing popcorn! * So this was an accident, or is it connected to the Twitter hack?

Will we take this as a much needed lesson about putting all of the internet's eggs into one basket? Probably not.

WebGazer.io managed to shoot me an email about my site being down. This in spite of their site being down too.

Hi @heliodor, WebGazer founder Gokhan here :)

Actually the site is running but not accessible due to the issue. Glad you got the heads up, after a while I had to pause monitoring to prevent side effects.

Yup. 1.1.1.1 stopped responding as well.

DNS resolution at 1.1.1.1 seems to have gone down and come back up for me in the course of 10-15 minutes.

Also breaking a bunch of deploys because npm and yarn are heavily dependent on Cloudflare, it would seem.

Remind me to check and see that I have 8.8.8.8 and 1.1.1.1 on my networks, not just one or the other..

It's back now (at least for me).

The uptime tool I use (StatusCake) is itself down... Was wondering why I didn't get an alert.

Seem to be localized issue. Cloudflare is up here in my country, but down for many people in US.

People talk about a single point of failure then go on to depend on a single point of failure.

The status page is now showing degraded performance for the Cloudflare API and Recursive DNS.

Cloudflare status site is also partially down. Some resources are not loading properly.

Of course, I read this after I spend an hour debugging some strange DNS issues.

Thank you for the early Friday.

I noticed Udemy was down when I wanted to go to the next video I was watching.

And now it's back up.

I can only see the dashboard down, all my sites with Cloudflare are up.

Looks like it's resolved -- we're coming back up at Repl.it.

Cloudflare DNS is down too

Is this US specific? Everything seems to work fine here in Europe.

Centralising on a single host suddenly not a good idea any more?

Seems like they managed to break half the internet for everyone.

how can I have cloudflare plus something else as a DNS failover? We are afraid to set a long TTL and have our IP changed for some reason. What do you guys recommend?

DNS is also (partially) down with my ISP (xs4all.nl) it seems

Ironic, isitdownrightnow.com is down.

All my DigitalOcean instances are down.

Yep, the DoorDash app is affected currently, my burritos!

Finally we see how much we depend on this single company.

Looks like it's starting to come back in the SE US.

Boats. People. Stop putting them in one single boat.

Back online for me.

Dns issues for sure

Well, sheit. This is all around the world. Press F.

Seems to work again for me in germany (Frankfurt)

League of Legends down too, not sure if related.

Most pages mentioned here seem functional again.

So THAT is why the Internet was acting weirdly!

DNS seems to be resolving for me in the UK now

Back online just now for me in Midwestern US.

2:36 PM PST - status.discord.com is back up.

NPM is down too.

Seems like it's starting to come back.

Was wondering why my DNS wasn’t working...

digitalocean.com DNS (on cloudflare) is now resolving again. looks like several things are coming back now.

I was having troubles with overleaf.com

How does Cloudflare compare to Akamai?

Ironically StatusCake is down as well.

who deployed on a friday afternoon?

Cedexis gets another lease on life

Jesus. Does anyone know anything?

Having issues with gitlab myself

Seems like it’s back right?

Can confirm for taskade.com

Friday

1.1.1.1 is back for me now

League of Legends, Valorant and Discord both down. I took today off to play games...

Felt like a BGP issue.

Looks like CF is up!

Some POPs are fine.

Five nines uptime?

Back online here.

the internet was built

to withstand a nuclear war

brought down by cloud flares

lmao it even took down my local stack

hn algolia search broken

yes hn.algolia.com is powered by Cloudflare /o\

more like Cloudflared

itch.io down

isitdownrightnow.com down

yes

Given that the US is basically in a non-shooting war with China, I wonder if this is something technical or part of some kind of attack. Something that I’d keep in mind.

There are enough ways for bits of the Internet to go kablooey on their own that “it’s an attack!” is a pretty big jump to a conclusion. If this turns out to be something other than Cloudflare tripping over a weird bug, my first guess would be that someone fat-fingered a BGP table yet again.

Your username is funny.

Reply on news.ycombinator.com