This reminds me of systems engineering and designing around recovering from inevitable failures.