Testing shell commands from Python
blog.esciencecenter.nl
blog.esciencecenter.nl
$ python3
>>> import sh
>>> sh.git.show('HEAD').count('\n')
49
# Now resize your terminal window to be 10 lines tall
>>> sh.git.show('HEAD').count('\n')
9
While you can append special flag for this, it is very easy to forget to do so, and the documentation does not help either. Their FAQ states that they are aware of the issue and won't fix it [0].
So my advice is: stay away from "sh" package, as it makes it very simple to generate unreliable code. Instead, use built-in tools which are pretty nice and just a bit more verbose:
>>> import subprocess
>>> subprocess.check_output('git show HEAD', shell=True).decode().count('\n')
49
sh provides clear semantics for certain use cases - that it doesn't match your expectations on how a program shold behave is understandable, but there are knobs to change that behavior, so I don't know if that is a strong enough argument to completely "stay away from" it altogether.
i know i've been happy it's there and used it in a few projects with good success, particularly projects that need exactly that: replace a shell and have the terminal passed down correctly.
In this light, I find the tty decision particularly bad, as it is very prone to introducing subtle changes -- for example, do you need the special flag for "git log"? what about "git for-each-ref"? what about "ls"? what about "gcc"?
For this reason, I find it way easier to just ban "sh" outright altogether. Our organization policy say "do not use sh module". When new member joins, and they way to shell stuff out, they will find out that they need to use subprocess module, grumble a bit, search the web, but then produce a working, if verbose, code.
An alternative would be to say: You can use sh, but never use it as sh's own front page recommends. If you find an article on the web which mentions sh, be aware that you cannot just copy-paste code from it, as it can be broken depending on which command you run. You need to audit which command you run to see if it cares about tty. Currently, you have to include the flag for many (but not all) git subcommands, "ls", "apt", ... but that list can change at anytime.
> i know i've been happy it's there and used it in a few projects with good success, particularly projects that need exactly that: replace a shell and have the terminal passed down correctly.
You have been lucky so far, but every time you use "sh" you are walking on landmines. I can give you a ton of plausible examples where sh breaks. Here is a shell replacement gone wrong:
$ git show HEAD | wc
49 273 2202
$ python3 -c 'import sh; print(sh.wc(sh.git.show("HEAD")))'
19 97 812
[0] https://docs.python.org/3/library/subprocess.html#subprocess...
You can define in a small test file what is the shell code to execute and what is the expected result, including error codes
I like sh very much in Python to generate scripts, but in terms of pure testing shell commands, I think cram is brilliant.
Here is an example on tests for a shell utility I created: https://github.com/jaimebuelta/ffind/tree/master/tests
The idea was to treat the shell process as a fixture, and conditions before and after the test (input file checksums, output files, generated stdout/stderr, etc.) would be checked. It was mostly for data analysis pipelines (hence the name).
I don't use it as much these days as my days of writing long running pipelines are behind me. But for that time, it did its job pretty ok. It was also a nice way to peek into pytest's internals.
Another commenter mentioned Cram, which I would look into if checking stdout/stderr (mostly) is enough for my use case.
If I am already using a specific framework to write the pipelines, I would also look into whether it already provides ways for testing.
I am not aware of other tools that test for pre-/post- conditions, so I would probably start with mine and see if need to tweak things along the way.
One thing I am not too happy about the plugin is that sometimes setting up the process as a fixture can be quite verbose. It feels like this part can be improved with some more thought.
More specifically, we were using the PBT-like `hypothesis` framework in Python for the purposes of more exhaustively exercising the CLI tools we were creating.
It worked pretty well.
``` def issue_command(self, command):
if not str(command):
response = None
logger.error('Command argument must be string')
else:
response = os.system(command)
if response != 0:
logger.error('Command returned non-zero status: {}'.format(command))
exit(1)
return response