Hacker News Reader

Comment by bpchaps | Hacker News Reader

An old coworker at a clearing firm once accidentally rebooted two very critical servers during a very critical time. From what I'm told, it ended up costing the company about a billion dollars in fees and fines.

He managed to keep his job.. just with no more prod access. :)

View on HN

Sounds like someone else’s fault, unless he owned and designed the system. No single engineer should be able to cause that kind of damage. Quorum rules, etc.

It was a team of five senior level linux sysadmins who oversaw about 2k servers. Lots of arguments for and against pushing blame here or there, but at the end of the day - he fucked up big time and shoulda known better.

I’m sure that’s true, but my point is that if you have that much money riding on a system you should have to figuratively (if not literally!) put two keys in and turn the lock at the same time to break shit. There should be systematically enforced mandatory reviews, two plus person policy for issuing commands, etc.

You have to expect people to make mistakes. I’m not saying he didn’t fuck up, but if a company is down a billion dollars the story should be of multiple people making multiple mistakes.

You're preaching to the choir. There's a reason the company was bought and completely gutted.

For what it's worth, a billion dollars for this company, while still a lot, isn't world ending. It was probably insured, too.

When people easily make mistakes like that. It’s usually the systems/process/design fault.

Hm. I mean yes, but still - the guy made a legit rookie mistake by not checking the hostname of the host he rebooted before typing "reboot". Kinda 101 stuffs there. :/

One imagines that if he were fired, he could have blown the whistle on the incompetent CIO/CTO who set up such a weak process.

A billion dollars in fines is an unfathomable mistake! It's on the level of a natural disaster, like a hurricane!

Would love to hear more about that :)

That's all I know, really. It was one of the strangest companies I've ever worked at.

Another story from the same place (won't confirm nor deny if it was the same sysadmin) - someone accidentally pushed a puppet config out that changed all hosts' timezones to CDT. A bit later, I received a random and kinda joking text from a buddy from a past job - who couldn't get an answer from my company's tech support - asking why all of their clearing reports had the wrong times.

He was the only person outside of three other sysadmins who ever mentioned anything was wrong. Not even my boss brought it up! It was odd.

What's a clearing?

Clearing (and settlement) is the process of delivering securities and receiving the cash after a securities transaction (ex: stock purchase).

Reply on news.ycombinator.com