Hacker News Reader

Comment by raverbashing | Hacker News Reader

This happens way more often than you think

A very simple example, you do something stupid on a remote machine (either high network usage or CPU usage) over SSH then you can't undo it because SSH becomes unresponsive

View on HN

Yeah got locked out of a dedi this way (bad ipfw ruleset which killed my ssh connection) and the virtual console wasn’t working. Fun times. At least it was a personal machine.

Is there anything that can be done to mitigate that? E.g. give ssh network and the daemon top cpu priority?

I'm a bit surprised there's no sort of SSH undo subroutine that reverses the previous command if connectivity is lost. Of course it couldn't cover every possible stupid thing but it could fix simple stupid mistakes like fouling up a port assignment or disabling the wrong network adapter.

A commit / confirm / rollback cycle is how this is typically dealt with in network automation.

How does ssh know what command you did and how to reverse it?

Recovery-oriented computing. Seperate Bohr bugs (easy to reproduce) from Heisenbugs (difficult to reproduce/change their behaviour once investigated).

Bohr bugs generate an alert and happily meander through normal support channels.

Heisenbugs go through phases -

1. Probation. On continued failure,

2. Restart. If the app or service fails after a restart,

3. Reboot. If the app or service fails after a reboot,

4. Re-image. If the app or service fails after re-imaging,

5. Remove/elimate the node.

Yes - use taskset or isolcpus with other magic to put sshd on its own CPU core, or one core per CPU. Lots of HFT places do that.

That doesn't help if the problem is a bandwidth congestion problem.

It can help some amount, though. Bind the NIC interrupts to a small handful of cores. Or, ensure that ssh only works through a management NIC, and have that NIC bound to the same cores as sshd. You can get really fancy with these setups, especially when working with NUMA stuffs

Some equipment will auto-revert to a last known good configuration if you don't approve new changes within a window... though high CPU could lock that process up..

in this case the old configuration was lost, it took an hour to rebuild because tooling normally used to rebuild it for testing was unavailable and building had to be done locally, using a single machine someone ssh-ed into, and that just takes a while. Luckily, a person was around who knew how to do the rebuild without fancy tooling.

Reply on news.ycombinator.com