Hacker News Reader

Comment by flowersjeff | Hacker News Reader

This really hits home, decades ago - I was working at this place that did daily tape backups. I remember thinking, this is unreal - there's literally a room filled with tapes.

One day, I asked if they ever had performed a recovery off of the tapes, as I questioned if the tapes were even being written to. (NOTE: Backups was not my job at all. )

Why had I brought this up? I would be in the server room and never saw the blinky lights on the tape...well.. blink. Everyone literally laughed at me, thought was a grade A moron.

A year later, servers died... Pop'ed in the tape... Blank. No worries, they had thousands more of these tapes. Sadly, they were all MT. They had to ship hard drives to a recovery shop, and it was rather expensive.

I left shortly after this.

View on HN

> Everyone literally laughed at me, thought was a grade A moron.

A note for anyone else in a similar situation - a good team doesn't ridicule someone for questions like these. A responsible leader should have cited a time in the past that they did a restore or a spot check, and no one should have laughed. The laughter sounds like masked fear or embarrassment.

This goes for any team. "How do we know this function of our job does what we think it does?" You should have an answer. Now, I've only worked in R&D software and not in IT. But IMO IT teams should work the same way in this regard.

> A responsible leader should have cited a time in the past that they did a restore or a spot check, and no one should have laughed. The laughter sounds like masked fear or embarrassment.

... or assigned the engineer asking questions the task of figuring it out!

Right? I regularly ask seemingly-rhetorical questions "just to make sure", and this approach helps me catch tons of otherwise-unnoticed issues. Being curious and vocal is a valuable approach in any technical business, IMO.

> a good team doesn't ridicule someone for questions like these

A good team won't ridicule any questions. If you're on a team that ridicules your questions, that's a huge red flag. Get out as soon as possible!

Yeah, they didn't even doubt themselves for a second, instead of challenging their own beliefs or at least showing the person that the backups were working before laughing.

Pop'ed in the tape... Blank.

Modern tape drives (like LTO) will at least do a read after write so you should never end up with blank tapes after a backup. But still no excuse not to do restore tests.

And make sure you're not storing your backup decryption key in the same backups that are encrypted with that key. Likewise, make sure you're doing restore tests on a "cold" system that doesn't already have that decryption key (or other magic settings) loaded, otherwise you may find out in a disaster that your decryption key is inaccessible.

That assumes that you're even doing the write in the first place, and not just logging a million "Error device not found" on your backup task. Speaking from personal experience, haha.

OP implies something like that is what was going on:

> as I questioned if the tapes were even being written to.

> I would be in the server room and never saw the blinky lights on the tape...well.. blink.

Reply on news.ycombinator.com