> Running administrative commands through SSH is just silly. With the tooling we have there's not even proper error reporting (how to tell that a command reached its target server but failed there? you can use heuristics of exit code at best).

The exit code of the remote process is exactly what you should use; I'm not seeing what you think is heuristical about it. Either you get back that the remote process failed, in which case you know it failed on the remote, or you get back success, in which case you know it succeeded, or you don't get anything back at all, in which case you just don't know. This isn't particular to exit codes per-se, any network message to a remote machine is going to be subject to the network or remote host possibly dying between sending it and receiving confirmation of whether or not it was received and had any effect.

If you're using SSH-the-process, it's going to mux things like network failures into that error code. You can either use a library (in a language a bit more robust than Bash) so that you know you're looking at the exit code of the remote process. If you must use SSH-the-process, you can always output the exit code of the remote process that you're interested in into stderr, in a manner that you can parse it back out on the receiving end.

> And if it was planned downtime? "Tough luck"?

Again, yes, though I don't see the "Tough Luck" about it; if I'm attempting to run some unknown and arbitrary command on a machine that was planned to be down, then yes, I need to step back and realize that as a human, and make sure that I'm doing the right thing. (Again, it depends on the command; what would be the effects of re-running it, and can you programmatically guarantee that re-running is a safe operation?)

> You'd much rather the server be monitored, with other systems being resilient than failing every now and then for random reasons and needing special additional actions to be taken to recover them.

Well, yes, I would rather the server be monitored. But monitoring is significantly orthogonal to the system I use to run administrative commands on the machine.

Again, it's not so much about being resilient as it is about not taking a bad situation and making it worst. I've run a lot of administrative commands across tons of machines that were not idempotent, and if any failed anywhere, I would need to intervene. (I also usually tested heavily prior to these, to make sure there was a decent chance of not needing to intervene later.) But that still, to me, implies that whether a given command should be retried until it definitely runs is at a layer higher than the tooling used to distribute commands out to host machines, and in the general case, could quite very well require a human.