Hacker News Reader

Comment by larrys | Hacker News Reader

It wasn't just a bad backup. It was a bad backup "system". One that didn't involve testing the backups obviously or even any redundancy. I guess the sysadmin never read the story (urban legend?) about the guy in Sweden who brought his backup tapes home and erased them on the car's electric seats.

The comment on the cartoon "I remember thinking this is really bad we're going to get our backups" comes from someone who obviously does not have the appropriate fear of what can happen to even begin to understand a good backup system. I'm guessing she is much more careful with her children and anticipates things in advance.

As far as "the backup being bad" where were they stored? Were they stored offsite?

View on HN

Quick tip: refresh your dev environments from the previous day's prod backups. You get to test them - and the people's skills to operate the backup system - every day, for free.

Probably written wrong not degraded.

It's very easy to say the backup should be tested, there should be mirrored recovery systems etc. but this is a movie, even in a studio system this is a one shot deal, the systems and even the hardware will be disposed of after the master is cut,the people will leave for other jobs etc.

Now in the middle of crunch to make the movie - tell the producers you are going to double the HW budget for a test system to test the backups and you are going to stop doing any new work until a backup system is built and tested.

Personally I'm amazed they had any sort of backup beyond personal copies on animators machines.

Probably written wrong not degraded.

That was my thought. Heck, it happened to me once.

Combination of a SCSI cable knocked loose, but not visibly so, and the brand new sysamdin (me in both cases) knowing enough to head and tail the backup log file looking for 'start' and 'complete' but not knowing to look in the middle for what was actually happening.

Lost a month of data in the (thankfully) little used legacy ERP system we used. If the entire plant had still been using it it would have been a disaster, not an inconvenience.

Pixar is a movie studio that developed mpviemaking technology over many years. No one is talking about backing up Tom Hanks in case he keels over.

"It's very easy to say the backup should be tested, there should be mirrored recovery systems etc"

I've been doing Unix backups for about 27 years. Back when it was more of a pain then it was in 1998 and certainly today. Total backup, incremental tapes the whole thing. Taking the tapes offsite. I wasn't a sysadmin but it was my business. So if I didn't have a backup it would be my problem and my loss both in work and money. I can assure you that setting up a backup plan (key word "plan") that evaluates the cost and trouble is worth it. The only reason higher ups don't approve of this is that they don't know about what can happen if something fails.

Ask yourself this question. Do you think Pixar changed their backup process after this event? I think they probably did. Not only because the failure happened to them but also because higher ups saw what could have happen if not for the woman who wanted to be with her child at home who saved the project in process.

Yep - There is nothing more eager to spend money on backups than senior management the day AFTER a disaster!

Reply on news.ycombinator.com