Hacker News Reader

Comment by paxys | Hacker News Reader

It was quite ironic that while every Facebook property was offline there was an immense amount of misinformation about the incident perpetuated across the internet (including right here on HN) which everyone just believed as fact.

View on HN

I didn’t see any disinformation, just initial reports that it was DNS which were later explained to be caused by BGP.

- It was government intervention

- Facebook was hacked

- They did it on purpose to bury the whistleblower story

- No one could access Facebook offices

- They had to cut open servers with angle grinders

- Disgruntled employees changed DNS records

- Lots of made up numbers for how much money Facebook/the rest of the economy was losing (or gaining)

They probably rushed out this blog post just to dispel some of these rumors.

I entirely believed that the badge readers on doors were broken, I have set systems like that up and they’re network connected. It is not the least bit surprising that BGP chaos would break that. That doesn’t mean people literally couldn’t get in the building, more like security was really annoyed and confused all day.

A New York Times reporter said employees had trouble getting in, and I don't see a retraction. So I believe that is true:

https://twitter.com/sheeraf/status/1445099150316503057

Do we hold journalists posting on Twitter to the same standards as writing in newspapers and their websites? Genuine question

Ha - journalists don't seem to have any standards for their websites so why would twitter be any different?

No, but I hold journalists posting on Twitter to a higher standard than someone unknown posting on Twitter. I don't see why she would lie.

Though, I assume by "had trouble getting in" means something boring like "was delayed by a few min until someone already inside pressed the exit button"

And they probably "fixed it" by putting someone near the door to let people in.

I briefly worked on access control a long time ago - aren’t they supposed to cache credentials locally so that individual doors still function autonomously during a fire / loss of power and comms?

Eh I would assume that would only be necessary if the lock mechanism kept people in, the only place I’ve been that did that was a defense contractor, and even then there were alarmed emergency doors. Everywhere else had either no exit control or trivially bypassed exit control.

Wasn’t defence, but somewhere with highly competitive commercially sensitive engineered components. You needed a card to open a door in either direction, but exit routes all had “Break to Exit” bypass switches - which lazy engineers often did, particularly if the door wasn’t covered by CCTV.

My curiously downvoted point was, I’m surprised an internet problem would stop an high security access control system from functioning - they’re supposed to be designed to cope with that and continue to work autonomously in emergencies.

> - It was government intervention

There was an Indian opposition Member of Parliament blaming the current government that it blocked FB due to some protests being held in the Capital.

Not to defend politicians looking to gain political mileage from everything, this is not a far-fetched claim. Current Indian government has blocked websites and shutdown internet access wholesale to towns and even whole states citing many reasons, and multiple times in recent past. Latest was internet shutdown in multiple Rajasthan towns apparently to prevent exam takers from cheating on a test.

The angle grinders stuff was confirmed by NYTimes. Everything else was rank speculation by people who admitted as such. No one was passing off a hostile attack as fact.

And then retracted by NYT:

> Correction: Oct. 4, 2021. An earlier version of this article misstated a Facebook team’s means of getting access to server computers at a data center in Santa Clara, Calif. The team did not have to cut through a cage using an industrial angle grinder.

Even if it wasn't true, drilling out locks or cutting them with tools is a pretty standard way of dealing with a lock you can't open. In an emergency where lots of things were broken it would be a perfectly reasonable thing to do. Also a news source with a nontechnical reporter might say the wrong thing that would indicate somebody was opening a server chassis with a powertool.

I think you mean repeated uncritically by NYT. It doesn't even make sense, as reported. Yes, maybe to get into the cage in a shared DC, but not individual servers, and it turns out neither was necessary. That's a rehash of an old Google story.

Another bit of misreporting (e.g. The Verge) is that this was "one of its main US data centers in California" when there's no such thing. The seriously big heaps of hardware are elsewhere and they're not shared, so there are no cages to force entry into. I've been in one; no cages in sight. I know which facility they're talking about, and its only distinguishing characteristic is that it's close to where relevant people live.

Most of these started out as speculation and jokes. Maybe we have a stupidity (i.e. critical thinking) problem rather than a misinformation problem. Only two of these (angle grinders and access to the offices) can really be considered lies or mistakes by an otherwise reputable source (the NYT) and one of those doesn't seem to have been refuted or retracted yet.

It looks like they retracted/corrected the angle grinder thing but they didn't use a retraction yet for the "access to office" info, maybe that means that was correct? (for a section of the FB workforce, of course).

You do know the meaning or jokes and/or ironic comments, right?

Just for the sake of completeness I saw a Twitter from some conspiracy website stating that it was some kind of menace from outer space.

It could still be any/some of these. Unlikely, but possible. Taking down BGP would just be a cover-up of something bigger. If this was a more honest company I'd call myself a tinfoiler, but since we're dealing with Facebook... ¯\_(ツ)_/¯

"They had to cut open servers with angle grinders"

I want to believe.

You say "believed as fact" where I say "speculated because we had nothing to go on".

Sure, my friends and I wondered if it was a malicious insider. It takes surprisingly few people in an organization to cause chaos.

Knowing IoT, it isn't unbelievable that badge readers could be offline.

Knowing division of duties, it isn't hard to believe that the network engineers, domain admins and datacenter ops people may have hustled to a DC to get things back online.

Never did I see a large number of people take anything as fact that didn't seem to be substantiated.

Seriously. I also saw lots of posts about how quiet it would be with Facebook down, but I don't think I've ever been exposed to so many stories and so much chatter about Facebook in a single day.

Do you think people on Facebook just talk about facebook??

I work in a different social media company that has had some visible outages. Its always hilarious to see how wrong people are with their confident speculation. It's a good reminder that people online are often full of shit.

I work in video games.

It amazing how wrong people can be and how confident they are about being right.

Even sometimes fighting _me_ about things _I_ designed and built.

It’s quite sobering; taught me not to believe all the speculation I read.

Just out of curiosity, how often do people claim that your random number generator is broken? And then when you ask why they think that, it's an anecdote about some time when they had incredibly bad luck?

it's more common for people to tell me the tickrate/framerate of my server or talk about how "[I] moved away from amazon to cheaper bare metal servers for cost savings" and such.

Lol I had the exact same lesson too. Saw people spewing falsehoods as facts and being misled by others. I’ve started to take everything I see online with a pinch of salt.

Dang I missed the misinformation.

The problem is not misinformation per se but the largest social media data processor running algorithms boosting such information for profit.

Like what?

We almost went down the ‘this is a subterfuge to delete whistleblower evidence’ rabbit hole.

I saw a couple of people clearly guessing something along these lines but none of them seemed to be claiming that it was actually happening, more like “isn’t it convenient that…”

The timing was uncanny. I still don't see a reason why it couldn't have been an intrusion/rogue employee? Like someone had access to a system to push router firmware updates or something?

A rogue employee would have been very easy to detect and that employee would have known this. The core network infrastructure involved is extremely sensitive and is quite unlikely to be accessible in a break in.

Also IIRC an employee was posting on Reddit saying the incident started shortly after a network update was posted this morning.

When you know more about the tech and systems involved a mistake seems infinitely more likely than sabotage.

Many have pointed out a couple of weeks ago Facebook released a paper talking about a new system to automate the management of the their BGP routing.

Seems like the new system having an unanticipated flaw is a far more likely scenario than a malicious actor.

More boring - but usually the boring stuff is the far more reasonable.

Did we, really?

There are a lot of comments promoting the idea here: https://news.ycombinator.com/item?id=28751224

Some of them seem to actually say it's likely, not just possible.

I could have believed it given Facebook’s record for unpleasantness, but I think the fact it must have cost them tens of millions in advertising revenue is evidence against this. Even the closest thing we have to a cyberpunk dystopia isn’t going to throw that much money down the crapper to bury bad press.

Lots of people blaming dns

I think mainstream news sources either used "DNS" as a term for routing internet traffic because most people haven't heard of BGP or come across autonomous systems (AS), or because they genuinely thought reports of routing issues meant it was DNS.

BGP took out their DNS. It isn't entirely wrong.

Reply on news.ycombinator.com