Working with Python subprocess | Hacker News Reader

Working with Python subprocess

jimmyg.org

41 pointsllambda12 comments

As someone who to this day still cringes every time they need to work with the subprocess module, do yourself a favor and use Kenneth Reitz's excellent wrapper library called Envoy:

https://github.com/kennethreitz/envoy

For reference, he's the same guy that did the Requests module, which is pretty much the defacto standard for avoiding urllib hell.

As someone who has used (and contributed code) to envoy, I must give warning that envoy is still very much a work in progress. Don't go in there expecting the same quality of the Requests module.

Envoy doesn't work if your data is larger than what you can safely store in memory (a few times over).

I would guess that subprocess is so ugly just because it needs to be so flexible. Personally, I don't find it so offensive, mostly because I have to take advantage of its "warts" on almost a daily basis.

A few notes from a recent attempt to use subprocess in the middle of a few generators:

p.communicate() only works if the output of the program can fit in memory. If it can't, python will gladly crash your box for you.

The buffer size of the os pipes on Linux are 64k. If you don't do your reads and writes in this size, it's possible to deadlock yourself.

Reads on the pipes has to be done in a non-blocking fashion, using os.read(), if you're dealing with streams of data that are larger than the memory you have available to yourself.

You can chain multiple subprocesses together directly, if you feed the stdout pipe to stdin on the next process.

    p1 = subprocess.Popen(['cat', 'foo'], stdout=subprocess.PIPE)
    p2 = subprocess.Popen(['grep', 'cat'], stdin=p1.stdout, stdout=subprocess.PIPE)
    p2.communicate()

select.poll() works on the os.pipes which are contained in the Popen objects.

I'm unmasking myself as a curmudgeon, but my blood boils when I see python people trying to do this stuff. The subprocess API is just awful. Compare the clean shell syntax at the top of this document to the wordy, awful dreck at the bottom required to do the same stuff in python.

Likewise compare it to the perl syntax that does the same thing (and happens to look at lot like the shell does, with good reason).

Python is a great language, but sometimes the aesthetic taste of its adherents just boggles me.

Edit: I see Pewpewarrows above pointing out that there's a third party module which fixes some of the nonsense in subprocess. And indeed, it looks fairly clean at first glance. So perhaps I should retract my blanket condemnation of its adherents' taste.

I agree. Scripting languages in general and Python in particular seem to be overused (and promoted) in domains that they're not well suited for.

But we've been seeing a lot of random several-year-old python stuff on Hacker News for quite a while now. Can't someone just post a link to a comprehensive list of all python tutorials, rather than them all being submitted individually like this?

I felt the same (well, not quite blood boiling, but annoyed) with regular expressions in python. Why so awkward? But now, I don't know, one thing the somewhat cumbersome subprocess and regular expression support helped me to learn was that I didn't want or need to spawn a subprocess or use a regex for every little thing. There are good python libraries for manipulating files and directories, globbing, and so forth. For strings, you can get a very long way using split, join, slicing, and the "in" operator. I like my programs more this way. They look like, you know, programs, instead of hacked together shell scripts.

Out of curiosity, what do you dislike in Python regexes? I switched from Perl to Python years ago and didn't notice much of a difference, other than that you have to write `value = re.sub(pattern, repl, value)` instead of `$value =~ s/pattern/repl/;`.

You have to write a lot more than that. Python's strings don't map well to regex syntax, forcing some odd quoting conventions and/or escapes. Python forces you to instantiate the regex at runtime instead of parse time, so if you want a compiled regex you need a separate bit of code to create it and manage its scope. Perl's captures pop out in automatic variables that can be directly used and interpolated, making simple parsing tasks (e.g. "print "$2 $1\n" if /.../") into sequences of easy-to-read one liners instead of page long "parsers".

All of that is the kind of thing that makes Python people's blood boil, of course, but it's absolutely the reason that regex handling is just so much natural in perl.

To answer the question "why would you want to do this in Python" - I have the simplest answer: Because the rest of my sysops code is in Python. And porting 400k lines of Python code to bash script (or Perl for that matter) is not worth a slightly cleaner (and potentially less flexible) subprocess framework.

I use subprocess on nearly a daily basis to automate database actions, and while the api is not the most appealing, it actually does its job very well, letting me coordinate 2-3 processes in a chain that's moving 100's of gigs of data on a regular basis.

In regards to "Accessing Return Values, poll() and wait()":

Another easy way to get a returncode without polling is to call communicate().

returncode will be set when it returns from communicate().

Article was written 3 years ago; perhaps things are better now?

Reply on news.ycombinator.com