Hacker News Reader

Comment by SMAAART | Hacker News Reader

>In addition to stranding billions of users, the Facebook outage also has stranded its employees from communicating with one another using their internal Facebook tools. That’s because Facebook’s email and tools are all managed in house and via the same domains that are now stranded.

SinglePointOfFailure.NoRedundancies.FB

View on HN

Having worked for a similar company, I remember there were some good old IRC servers up and running to communicate in an emergency just like that.

Facebook has this too, but they require facebook.com DNS to work, so they are also down.

Nice try, but they have separate communication channels for SREs so don't worry.

You mean they don't receive all their alerts through facebook messenger?

Thanos snapped his fingers and Zuckerberg vanished with the keys.

My (very large) employer had a worldwide outage a few years ago where a single bad DNS update stopped everything in its tracks (at the time many things were still in our own data centers, now more is in Amazon/etc). It took most of the day to restart everything. But it's not something most people would have noticed like FB. Thankfully I worked in mobile so not involved.

It is hard to balance dogfooding (good) with SPOF (bad), many big companies do get it wrong (AWS with S3, Slack in the recent past) all the time.

It is easy to get it wrong if your company provides internet services that every developer typical depends in their workflows and to keep educating your own developers on how to use them and when not to use your own services.

Reply on news.ycombinator.com