Comment by dozzie | Hacker News Reader

Comment by dozzie | Hacker News Reader

Parent Full thread

> The exit code of the remote process is exactly what you should use; I'm not seeing what you think is heuristical about it.

Lucky you if your SSH client never failed to connect. How do you think it reports a network failure?

> This isn't particular to exit codes per-se, any network message to a remote machine is going to be subject to the network or remote host possibly dying between sending it and receiving confirmation of whether or not it was received and had any effect.

You see, there are tools that allow you to easily tell between network failure and remote command failure. RPC protocols are an example.

> If you're using SSH-the-process, it's going to mux things like network failures into that error code. You can either use a library (in a language a bit more robust than Bash) so that you know you're looking at the exit code of the remote process.

Now let's get practical: find me such a library.

>> And if it was planned downtime? "Tough luck"?

> [...] if I'm attempting to run some unknown and arbitrary command on a machine that was planned to be down, then yes, I need to step back and realize that as a human, and make sure that I'm doing the right thing.

We were talking about configuring servers. You basically tell me that I would have to pay for "agentlessness" with tracking state of each and every machine I would possibly want to reconfigure, and remembering to check if the machine was booted to re-issue a reconfiguration command.

> (Again, it depends on the command; what would be the effects of re-running it, and can you programmatically guarantee that re-running is a safe operation?)

Sorry, I'm lost here: what re-running? The command didn't get the chance to run in the first place. I don't know where did you get the idea that I propose running the command again if it got through to the remote server and failed there.

> Now let's get practical: find me such a library.

Certainly. I'm most familiar with Python, and for that, both Paramiko and asyncssh support returning the exit status of a command, distinct from any network failure.

As I mentioned in the above reply, even with command-line ssh (and any other library, really), if you're creative about how you structure the output on stderr, you can encode the exit status there to be able to differentiate it from ssh returning other errors. E.g., if you pipe the entirety of stderr through, say, base64 and then append at the end a newline and the exit status, you can parse out the exit status on the receiving end. (This is admittedly quite a hack: using a library in Python with direct access to the data is much nicer. The ssh binary is mostly a porcelain interface to the SSH protocol; libraries are a better tool for building on top of.)

> with tracking state of each and every machine I would possibly want to reconfigure, and remembering to check if the machine was booted to re-issue a reconfiguration command.

Perhaps this is just a difference in our production environments. We strive very hard to have all our machines online at all times. (Excepting short outages for rebooting for patches, etc., of course.) We use VMs, so there's not much point to having it be shut down.

> Sorry, I'm lost here: what re-running?

My larger point was about non idempotent commands. If your infrastructure retries commands, are you just very careful to not have non-idempotent stuff? Generally, these things are either "run at most once" or "run at least once"; running exactly once is (in general) impossible, as the command might finish successfully, but the host machine crash prior to recording that success. Again, if you can structure the command to be idempotent, you can work around this, but in my experience there always seems to be some patch or database upgrade that this can't be accomplished for.

>> Now let's get practical: find me such a library.

> Certainly. I'm most familiar with Python, and for that, both Paramiko and asyncssh support returning the exit status of a command, distinct from any network failure.

Oh, that's something new. From what I heard, Paramiko used `ssh' under the hood, but apparently I was wrong.

> ([...] The ssh binary is mostly a porcelain interface to the SSH protocol; libraries are a better tool for building on top of.)

There's still a big problem on the client side itself. SSHv2 is a complicated protocol, so client libraries' APIs are not pretty or force you to use OpenSSH-like home (or both, like Paramiko does).

>> with tracking state of each and every machine I would possibly want to reconfigure, and remembering to check if the machine was booted to re-issue a reconfiguration command.

> Perhaps this is just a difference in our production environments. We strive very hard to have all our machines online at all times.

You must then have very homogenous environment with a small team. Sounds like a startup of sorts. Anything more loose (e.g. more than one team working on the servers, or split between who installs the machines in racks and their OSes on disk and who manages the OSes) and you won't have the luxury of keeping everything up and running all the time.

>> Sorry, I'm lost here: what re-running?

> My larger point was about non idempotent commands. If your infrastructure retries commands, are you just very careful to not have non-idempotent stuff?

And what does it have anything to do with re-sending the commands when they didn't reach their target machine?

> Again, if you can structure the command to be idempotent, you can work around this, but in my experience there always seems to be some patch or database upgrade that this can't be accomplished for.

Uhm... What? Where did you get the idea that upgrading database is managing configuration? It's a deployment operation. A different category.

> And what does it have anything to do with re-sending the commands when they didn't reach their target machine?

It has to do with whether its at all safe to run the command twice. If you don't hear back from a machine that your command ran, you don't know if the command reached the machine or not, in general. If you decide to not re-run the command, it may be that it never made it, and you never run the command. If you decide to re-run the command, it may have reached the machine, and you've now run the command twice. Consider:

  1. You send command $X to a remote machine.
  2. The remote receives and executes $X successfully.
  3. Just prior to sending back that $X was successful, the remote dies.

From the point of view of the deployment machine, you never hear back from the remote. You don't even need two machines to get into this state: even on a single command, if you run $X and the machine crashes just as $X is about to return whether or not it succeeded or failed, you're out of luck.

For specific commands, such as ones that log their progress and can recover from crashes, you can determine after a crash the correct procedure automatically. But not every arbitrary Unix administration command falls into that bucket, especially in the weird and wacky things that need to be done to keep production machines alive.

> Uhm... What? Where did you get the idea that upgrading database is managing configuration? It's a deployment operation. A different category.

We used our deployment infra for all sorts of fun stuff.

Again, thanks for the objections, they're making me think hard about the design choices.

> Lucky you if your SSH client never failed to connect. How do you think it reports a network failure?

Currently posixcube handles this by checking the return code and reporting the error.

> You see, there are tools that allow you to easily tell between network failure and remote command failure. RPC protocols are an example.

ssh does have some basic differentiation: "ssh exits with the exit status of the remote command or with 255 if an error occurred." Although, of course, 255 might be returned by a remote command (not sure how likely that is), so it's still ambiguous. I guess you're saying that if it's an SSH error, then that's pretty safe to retry and it would be a nice feature, and I agree so I've opened this issue: https://github.com/myplaceonline/posixcube/issues/6

> Now let's get practical: find me such a library.

I guess it would just be a matter of, first, checking for 255, and next, checking the output of the command to see if it's an SSH error for sure?

> We were talking about configuring servers. You basically tell me that I would have to pay for "agentlessness" with tracking state of each and every machine I would possibly want to reconfigure, and remembering to check if the machine was booted to re-issue a reconfiguration command.

So you bring up a good point. On DigitalOcean, for example, their Fedora images use extlinux for booting and I can easily update the Linux command line (e.g. crashkernel), reboot, and it picks it up. Their Ubuntu distribution uses grub, and update-grub works fine, but for some reason, it requires a hard shutdown instead of a reboot to pick up the change. The way I handle this now is here: https://github.com/myplaceonline/myplaceonline_posixcubes/bl...

We echo this instruction to the user. This prints a red-text error message and stops execution. This seems even better than an agent-based execution because with an agent-based architecture, the agent is now gone (hard powered off); whereas, this style of orchestration in the front-end can use a script on top of posixcube.sh that waits a minute, remotely calls DO to start the droplet, and then continues execution (using the same script since it's essentially idempotent). The only other option would be to have a centralized server that orchestrates all of this, which posixcube avoids.

> Sorry, I'm lost here: what re-running? The command didn't get the chance to run in the first place. I don't know where did you get the idea that I propose running the command again if it got through to the remote server and failed there.

See above, I'll need to try it, but I think the ssh error code for "command didn't get through" along with some simple heuristics on the command output /might/ work; I'll investigate in issue #6.

> ssh does have some basic differentiation: "ssh exits with the exit status of the remote command or with 255 if an error occurred." Although, of course, 255 might be returned by a remote command (not sure how likely that is), so it's still ambiguous.

If one uses SSH for running such scripts, then it is not unlikely that the script itself would have another call to `ssh' to an even-more-remote machine.

And then there is the case of not expecting that a command would ever exit with 255 code. My trust to a system would be quite damaged if I happened to run into such a bug that a proper remote error would be mistaken for a network error. How can I, as the script writer, be sure that I never run into this situation?

I agree the ambiguous return code is a major limitation of SSH. I think I'll just avoid any retry logic and treat every failure the same: fail immediately and let the user to decide how to retry. Since posixcube does most of its execution over a single SSH connection (packaging all commands/cubes into a cube_exec.sh which is executed remotely), I think the immediate need for automatic retry on failing to connect over SSH isn't large.

Reply on news.ycombinator.com