What about off-site backup? How many failures can cascade in a way to wipe out off-site backups?
What about off-site backup? How many failures can cascade in a way to wipe out off-site backups?
Or not even offsited, it's not like the building burned down. Just tapes not left in the library.
The theory that they might have the data but just don't know how to restore things because all the Danger people are gone, now that actually seems somewhat more credible to me.
Because SANs allow multiple machines to write to the same disk, they have byzantine failure modes where data can be overwritten on disk, but cached for long periods of time by the machines that need it so there's no visible problem until after a power failure.
When data is migrated to new database hardware, sometimes the backups haven't actually been tested. Sometimes during the cutover, someone who didn't fully understand the architecture backs up the new (empty) machine on top of the old machine's backups, so there is no backup for a while. All these mistakes can happen pretty easily when sysadmins are woken up in the night to fix a service outage while people are yelling at them. I have made some of these mistakes myself, though so far I've been lucky.
The computer industry has never learned to admit that there is always some level of risk. You won't hear an oil company exec say "Inconceivable!" when an iceberg crashes into a North Sea oil rig, or a refinery catches on fire. Systems fail, and all you can do is try your best to avoid it and move forward after it does happen.
Some of the things like 'locking the keys in the car' can be mitigated by making sure that you test you backup system.
Obviously you can't prevent everything, but to say that some of these issues are 'unavoidable risk' is like saying that an oil company exec would say, "shit happens," to reports that one of their oil tanker captains crashed a ship while drunk.
The real problem is that backup is usually an after-though.