One idea I had was to enable something annoying and kooky like port-knocking or OTP pass/connection enabled 'backdoor entrance' for emergencies, but ended up being too lazy, and realised it was just expanding the attack surface.
To my original point, we had an interesting setup for one network where the firewall changes were all manipulated via a script which required the change to be applied, and then confirmed after a short delay with a different command, otherwise after 5 minutes the ruleset would revert to the previous known-working one. It definitely saved some downtime/late night DC trips.
$ ssh -A -t office_user@office ssh production_user@production
I have the same private key in all my machines, so this worked transparently avoiding the IP restriction of iptables of the production machine.