The best case scenario of a bunch of engineers flailing about on a bridge turning knobs is that you luck into a fix but don't know how you got there. But you're more likely to make things worse.
The best case scenario of a bunch of engineers flailing about on a bridge turning knobs is that you luck into a fix but don't know how you got there. But you're more likely to make things worse.
If all the alarms are going off because of a loss of redundancy, then currently there is no outage. The correct move should be carefully considered, and maybe tested in the sandbox environment.
If there is currently a 100% outage, it's best to go all out on trying every possible fix, because typically you'll restore service quicker that way. Sure, occasionally you dig yourself a deeper hole, but usually it's the best strategy.
Maybe. Once an outage has happened, an additional 30 minutes or hour of outage, depending 100% on the service in question, might be a bearable cost if it means preventing a domino effect of future issues caused by measures taken to restore the outage in a hurry.
Almost assuredly not. If a system hits a 100% outage, there are about to be a series of cascading failures by dependent systems. If you don't even understand which system is the root cause, all you're doing is testing a bunch of never-before-tried combinations in production and hoping something works.