Hacker News Reader

Comment by mkrecny | Hacker News Reader

Fresh out of college I was responsible for a WFIO at a 40 person company - ultimately had to leave.

http://edu.mkrecny.com/thoughts/how-i-fired-myself

"Um those [backups] got really expensive, so we stopped doing them about a month ago" sounds painfully familiar.

I remember reading a while ago and cringing at this so bad, this needs to be a cautionary tale of version control, development/staging/production environments and ... backups.

"The Startup CTO's Guide to ..." ?

all it contains is the word: "backups".

Multiple times, for redundancy.

It should probably also contain the word "documentation of system configurations".

But why would a rockstar-ninja-pirate ever need backups?

Backups are not MVP! First to market!

I got cold sweats reading that story. It's a shame you had to pay the price for the abysmal operating practices of the company.

Thanks, but ultimately, leaving when I did was extremely positive. I ended up in NYC in the employ of Techstars which turned out to be a much better place for me.

FWIW, whoever built a system that had no redundancy or disaster recovery plan or even properly separated environments was responsible. You were just unlucky.

I did something similar about 10 years ago at my first dev job. I forgot a "WHERE" in an update and destroyed every customer in the live db.

The issue was that my manager never caught it in my migration file and it went live.

I was lucky that we backed up every day..and didn't really lose a ton of data.

I never really understood why databases don't have undo.

I largely avoid UPDATE and DELETE statements. Changes are new rows, time ordered, which supersede prior rows. Like an audit log.

Works great. Especially for anything that needs history, e.g. medical data.

Wrote one of those many years ago, yes - for medical data. It's the right solution to just about every problem actually, and I'm surprised that it isn't more commonly taught.

Yeah, logical deletes ftw :) .

One thing I haven't figured out is how to archive old records, I think it's not a problem for modern DBMSs, but we have a problem at work with 15 years' worth of historical data.

They do: http://www.oracle.com/technetwork/database/features/availabi...

We persuaded mySQL to add the --i-am-a-dummy (http://sql-info.de/mysql/notes/I-am-a-dummy.html) option after about the third time one of our developers did this.

You may have been the straw that broke the camel's back, but you shouldn't feel bad about that. I recently had a junior sysadmin learn the hard way about BEGIN TRAN/ROLLBACK/COMMIT. But that's all it was, an embarrassing lesson as he explained to a customer why they'd need to wait a little bit longer.

Even a major "DELETE FROM Foo" should just immediately get the DBA switching to read-only mode, and then restoring using the transaction logs. It's annoying, it causes a bit of downtime, but it's not a "Big Deal".

Yeah, that would be a nasty mess for us too... the change would replicate to all our databases and we'd have to stop the world, restore from the previous day's backups, re-run all the database binlogs up until the point where the disaster happened. It would be a couple of hours of total downtime - not a happy place.

But we do have daily backups and binlogs in multiple places, so it wouldn't actually lead to data loss. It really sounds like you were lucky NOT to be there!

For some reason this is one of the worst of these I've ever read.

I've personally f'd up pretty bad restoring a database to the wrong the site and blowing away 3 months of content. We didn't ever do any offsite backups and I just lucky to have an in with our hosting company and they managed to get everything back but those 3 months.

Once bitten, you're always way more cautious but it's easy to get lazy again from time to time.

Great story, I remember reading it the first time around. How they didn't explode sooner from that set up and and managed to get as far as they did was probably the most shocking part of it.

This is why as a dev I never want anything more than read permission with the production environment.

Reply on news.ycombinator.com