Hacker News Reader

Comment by rbanffy | Hacker News Reader

I had a RAID-5 array of five identical, same batch and close serial numbers. One went bad. I ordered replacement and procurement blocked the purchase until I got it approved by my manager. I warned it was urgent. They delayed it for 3 days. On day 2, the second drive died.

We had backups and could replay all db transactions up to the minute. Still, the service stopped for more than 24 hours. It took a week to clear the backlog.

View on HN

Delayed approval of an urgent 3-figure purchase resulting in a 5-6 figure loss in productivity, customer sat, etc. Great example of penny-wise, pound-foolish!

Indeed. Small powers.

Buying individual drives to shave a few pennies is pretty foolish to begin with, so why stop there!

Make it an OEM problem.

In their defense, we had a bunch of drives coming in in a month. It's just that we ran out of spares faster than expected (we had to increase capacity in a couple servers on short notice).

Hopefully they let you keep at least one new drive around as a backup after that experience. If you're not running RAID6 you should at least have one on the shelf, preferably from a different batch.

That was in 2003 or so. IIRC, mdadm didn't have raid6 support back then and the chassis wouldn't support more than 4 devices.

Those were the heroic days.

Honestly though, 1) the array rebuild might take that long time anyway; 2) the stress of the array rebuilding would probably make the other drive fail anyway.

Reply on news.ycombinator.com