Hacker News Reader

Comment by downandout | Hacker News Reader

Agreed. How could a company with "millions in revenue" not backup critical databases? Not only were they exposed to the threat of human error, but hardware failures, hackers, etc. When he submitted his resignation, the company should have encouraged him to stay. Instead, anyone at the company that had anything to do with the failure to implement regular database backups and the use of redundant databases should have been fired.

I've had more than my fair share of failures in the start up world. It always drives me crazy to see internet companies that have absolutely no technical ability succeed, while competing services that are far superior technically never get any traction.

View on HN

It's the difference between understanding the incident (programmer makes mistake) and the root cause (failing at data integrity 101). Not just no backups, but no audit log of what's happened in the game - I'd expect an MMO to have an append-only event history quite apart from their state information.

At least all transactions for purchased game items should have been logged in a separate database. There's nothing worse for a digital content company than forgetting who bought what.

And worse, they had a backup service, and then dropped it to save money.

This sounds so unreal that I am having doubts about the veracity of the story and would defer any judgement before hearing from the "other side".

Unfortunately it sounds perfectly plausible to me.

"Why are we paying for backups? The database has never failed (yet)"!

Super important goal my boss gave me: reduce expenditures...CHECK! Can't wait for that bonus check!!

Having just helped a friend who was miffed at the idea of spending money on a new UPS for >$5k worth of networking equipment: do not be surprised.

Penny-wise pound-foolish.

Spending $5k on a UPS is very different from not having backups of your production database which runs your multimillion dollar business. This story just doesn't add up.

You misunderstood the comment. Angersock is saying that the networking equipment cost more than $5k and the friend was unwilling to buy a UPS to protect the equipment.

I've seen something like this. It could be a situation where they were transitioning from that backup service to their own server or another service, and the move was never completed due to some hiccup or priority change.

But if no one is watching to ensure the move was finished (or they got distracted), then something many people treat as set-it-and-forget-it could easily get into that state.

If you are transitioning from one backup service to another, shouldn't you only cancel your old service, after you successful set up and tested the new one?

You would be amazed if I told you how many "vital" services are like the blog writers former employers: One button push from destruction.

Oh? Tell me.

Sounds completely possible to me - I have worked in a lot of environments and at some places I've seen decisions that make cancelling backups look like an act of genius.

One interesting observation I can make: no correlation between excellence in operations and commercial success!

I don't think it is that far-fetched. I've had experience with a well known company that does millions in revenue per week (web based shopping cart) that just FTP's everything with no DVCS. Designers, developers, managers all have access to the server and db.

Reply on news.ycombinator.com