>"The rule of thumb is when something is working you don't change it," Hertz said. "We'd like to change as few things as possible when we bring Hubble back into service."
The best case scenario of a bunch of engineers flailing about on a bridge turning knobs is that you luck into a fix but don't know how you got there. But you're more likely to make things worse.
If all the alarms are going off because of a loss of redundancy, then currently there is no outage. The correct move should be carefully considered, and maybe tested in the sandbox environment.
If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy.
Maybe. Once an outage has happened, an additional 30 minutes or hour of outage, depending 100% on the service in question, might be a bearable cost if it means preventing a domino effect of future issues caused by measures taken to restore the outage in a hurry.
Almost assuredly not. If a system hits a 100% outage, there are about to be a series of cascading failures by dependent systems. If you don't even understand which system is the root cause, all you're doing is testing a bunch of never-before-tried combinations in production and hoping something works.
Business requirements and requests change all the time. 90% of our work is done in response to that. The other 10% is fixing up technical problems due to increased scaling or bugs found, and then basically never do we upgrade a system to keep up with security updates or change to a more modern tech.
Sure, there's often a reason like "we wanted to write it in a different language" or "we've overhauled the UI to be slower and more cumbersome".