Server with Full Hard Drive Shuts Down Every Toyota Plant in Japan

thedrive.com

8 pointsperardi3 comments

> Like any diligent organization, Toyota had a backup in place, but as it was running "on the same system,"

The classic “let’s back up our hard drive to the same hard drive”, I’m guessing.

Its hard to understand explicitly given the light level of detail in the article, but it seemed like they had a redundant system to take over in case of hardware failure.

Unfortunately due to the nature of the underlying issue, the hot spare suffered an identical problem (ran out of drive space) given it was identically spec'd.

Good example lesson to show that simple redundancy doesn't make all problems go away in the case of failure.

“Disk full” isn’t a hardware failure though, it’s a failure of monitoring and management, which are aspects of solution design you can’t reasonably specc away.

Reply on news.ycombinator.com