Python for Humans
python-for-humans.heroku.com
python-for-humans.heroku.com
The back story is that the older APIs that Python comes with -- os.popen and os.system -- are deprecated. Programmers are urged to use the "subprocess" module instead. Although this doesn't have the problems of the original functions, it has a rather arcane interface, in particular if you want to read the output (stdout or stderr) of a subprocess.
"envoy" seems to aim at fixing this, by providing sane defaults and being optimized for the common case. However, these defaults have drawbacks of their own.
1. envoy defaults to keeping the process output in memory, as a giant string. This can be a bad choice with regard to memory usage and performance.
2. You can run several processes in a pipe using ("cat foo | grep bla"). But otherwise as far as I can see, run() ignores regular Shell semantics, such as quotes. I imagine this can lead to unexpected results. The amount of data passed from one process to the next is capped at 10 MB -- recipe for bugs that are hard to find.
3. subprocess.call() accepts an array in the style of ["ls", "-l", "/mnt/My SD card"]. This has obvious advantages over having to deal with escaping shell characters. A good API should preserve this advantage over os.system().
4. The defaults cannot be overridden, and no preperations have been made to allow changing them. Of course this can be changed in the future. However, one of the reasons the subprocess.* API is convoluted is that it allows all kinds of flexibility, much of which is needed in many serious programs. It may be difficult to add this flexibility to envoy at a later stage. The point is that a flexible API is hard.
None of this is to discourage this initiative, which seems to me a much-needed improvement over Python's built-in API. Also, with a version number as low as 0.0.2, there is probably little need to worry about API compatibility.
oputput = subprocess.check_output(my_command)Unless you're running on Windows, in which case IME it will corrupt your carefully constructed parameters in completely inappropriate ways that can be debugged only at the cost of (a) changing the call() to execute a script that dumps the actual parameters supplied verbatim, and (b) at least an hour of your life that you're never getting back.
This "feature" is about one step above MS Word's default autoreplace behaviour in irritation level. What happened to "Explicit is better than implicit" and "Special cases aren't special enough to break the rules"?
I seem to remember that on Win32, all you get is an argument string, and the process is required to do the parsing itself. This is simply a different model, and the Unix way is cleaner and easier to work with. It seems to me reasonable to optimize for the saner (Unix) way of doing things instead. I confess I don't understand your analogy to Word.
Neither of these approaches is inherently superior, they're just placing responsibility for certain operations in different places. But the fact is that because Windows programs don't have to assume a certain set of conventions for their command line, many do not, and if you have the misfortune to want to automate those using subprocess, you're in for a world of pain (until you just give up and use the single-string version instead of the list of arguments, having realised that this is enough to stop it messing around with your carefully crafted strings and just pass them through verbatim).
edit: If anyone wants to see a real-world refactor from HTTPLib/2 to requests, I did so with Pysolr here: https://github.com/mattdeboard/pysolr/commit/db63d8910dec42d...
I had to highlight the text and drag downwards in order to see the content. But it was annoying having to do this for every slide with a lot of content.
Otherwise, these libraries seem really useful. Thanks for this.
Whats so wrong with just sticking a bunch of static slides on a page one after another?
I'd like to see some community effort to build a collection of similar "better than the standard" libs.
Which, at some point, could replace the standard libs. Or be the de facto standard, a pip install call away...
Libraries tend to move more quickly than the language and interpreter/compiler. Tying them together, while convenient, often leads to rot, clunky libraries, slow moving updates, and libraries being build to the interpreter/compiler instead of to the needs of the users.
I would like to see instead a somewhat canonical (widely accepted) list of the highest quality libraries for a given set of needs, with information and pro/con/caveats listed for each, instead of them being included in the mainline trunk.
I really applaud what Kenneth Reitz has been doing lately.
-- a very happy user of the `requests` library
And those functions are more convenient than the default behavior of backticks, because they handle for you raising an exception if the subprocess fails.
Python:
import subprocess
output = subprocess.check_output('command')
$output = `command`
die "failed: $output" if $? from subprocess import check_output as qx
output = qx(['command', 'arg1'])If you have a look at the older libraries, most of them were written in a procedural style. Not only that, it is very amenable to testing in the REPL.
import smtplib
s=smtplib.SMTP("localhost")
s.sendmail("me@my.org",tolist,msg)
If you have a look at the XML library, roughly when "patterns" became popular, this style of thinking infested standard library contributions. It also coincides with a time when camelCased function names crept into the python standard library.
Here's one in xml/dom/pulldom.py:
self.documentFactory = documentFactory
Here's more pain from libraries of the same era, some of these I'd argue un-Pythonic:
#xml/sax/xmlreader.py:
def setContentHandler(self, handler):
#wsgiref/simple_server.py:
class ServerHandler(SimpleHandler):
#urllib2.py:
class HTTPDigestAuthHandler(BaseHandler,
AbstractDigestAuthHandler):
Even logging is infected by the same over-patterning. logging/__init__.py:
class StreamHandler(Handler)
LoggerAdapter(someLogger, dict(p1=v1, p2="v2"))
#multiprocessing/managers.py:
class BaseManager(object)
Initially, buttons had to be subclassed just to handle click events, since functions were not first class objects. Then someone came up with a MouseListener interface, which proved too unwieldy to handle a single click. So the MouseEventAdapters came into being.
Therefore, to handle a click in a "pattern" manner involves
an anonymous class
which subclasses MouseAdapter
which implements MouseListener,
which overrides MouseClick.
Publishing how industry solves this problem of "MouseClick" over and over as a pattern [design pattern is a general reusable solution to a commonly occurring problem within a given context in software design] only gives legitimacy to an approach that has dubious wider applicability.
Heavens help the future developers who are forced to do it because it is now recognized as being industrially "good practice" and codified in a reknowned book.
It isn't!
It was a style that was forced by the constraints of a language.
This is neither pythonic nor necessary:
panel.addMouseListener
(
new MouseAdapter ()
{
public void mouseEntered (MouseEvent e) {
System.out.println (e.toString ());
}
}
);
def mouseEntered(event):
print event
panel.mouseEntered = mouseEntered
panel.mouseEntered.append(mouseEntered)
>>> import log
>>> log.log(log.CRITICAL, 'PrintModule', 'Printer failure', sys.exc_info())
PrintModule: Printer failure
>>> print log.configuration
[<function printToStdErr as 0xcafebad>]
>>> log.configuration.clear()
>>> print log.configuration
[]
>>> def send_mail(level, module, message, exception):
... import smtplib
... smptlib.SMTP('mail.test.com').send('syslog@test.com',
... 'no-reply@mysystem.com', module + ' ' + message)
...
>>> log.configuration.append(send_mail)
>>> log.log(log.CRITICAL, 'PrintModule', 'Printer failure', sys.exc_info())
etc.
>>> import log
>>> log.configuration.extend([
... log.sendmail,
... log.syslog,
... log.rotatinglog])
...
>>> log.sendmail.to = 'admin@test.com'
You might be wondering how to assign an attribute to a function, like sendmail.to?
In this particular case, sendmail is actually a class that masquerades as a function.
>>> class Sendmail:
... to = None
...
... def __call__(self, level, module, msg, exc):
... # et. c
...
>>> sendmail = Sendmail()
log.getLogger()
def foo(): pass
foo.a="asdf"
>>> foo.a
'asdf'
>>> foo.__dict__
{'a': 'asdf'}
or alternatively:
>>> def bar():
... bar.a = "asdf"
...
>>> bar.a
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
>>> bar()
>>> bar.a
'asdf'
>>> def f():
... print f.something
...
>>> f.something = 'blah'
>>> f()
blahTherefore to handle a mouseClick event requires a class. However, it doesn't make sense to have a IMouseClick interface that has only one method, right? That would seem too wasteful.
So the natural thing to do is to group all the mouse events into a single interface.
Before you know it, you have the abomination of MouseClickListeners, MouseAdapters, and anonymous classes.
About overusing patterns - GOF is really a book of anti patterns that is particularly useful for language designers, because it points out the glaring contortions that people have to do to get their work done.
On the other hand, being a good programmer requires learning and subsequent unlearning of GOF patterns. I'm reminded of Bruce Lee when he started with Wing Chun, but then found the "form" too rigid. Quote from http://www.cavendishscience.org/phys/brucelee/brucelee.htm
In describing his style, Bruce Lee made a clear
distinction between "having no form" and "having no
form." To have no form was to transcend form from a
position of mastery that enabled one to see when the
rules could be-in fact, needed to be-broken. No form
was highly disciplined in its execution. In essence,
to have no form was to have access to all forms; to
understand all forms at their most essential level
and to see and be able to act on the connections among
forms. On the other hand, "having no form" implied an
undisciplined approach, an inability to master any form.
Lee discovered that he could not teach no form directly
to novices in the martial arts. A student first had to
master the form of karate, tae kwon do, or some other
school; only from a position of mastery could the
student begin to experiment with abandoning the form. > If anyone remembers, Java had to do OO in a
> big-style with OO everywhere -- there were no
> alternatives.
Related to what you're saying about the GUI, I'd be interested to see a detailed summary of what the Lighthouse people did, and how it was different to Java. I've found that NeXT tradition stuff - despite claims that it's heavily OO - in facat tends to err away from subclassing towards composition. I suspect the Lighthouse interface patterns did too.
In Cocoa touch, it's:
[mybutton addTarget:someObjectMaybeSelf action:@selector(onClick) forControlEvents:UIControlEventTouchupInside];
The framework usually use delegates when a data source is needed. (for example to build up a table, you need to specify how many rows, what goes into each row etc).
Cocoa touch is really clean. Cocoa is still nice but a little messy.
I agree with the author's goals of making common tasks easier and more obvious. urllib2 is an easy target, as it was added to the standard library over a decade ago, long before REST was something people talked about. The best tools for packaging, versioning, and testing have always been a bit ambiguous in any language, including Python.
However, the author points out something that has always bothered me about Python: it is way harder to start a subprocess with an external command in Python than almost any other language. This has been true whether using sys or os or even subprocess, which is quite recent.
I always felt that this had something to do with the constant warnings in the documentation about how a pipe between the subprocess and the Python process might fill and cause the subprocess to block. Or how running the program through shell rather than exec or something might cause some sort of security issue. Are these real issues that other languages ignore in the name of user convenience, or has Python just never been able to make the right API (as the author seems to argue)?
There's no fundamental problem that's stopped Python from doing this before. For some reason, all of the ways to spawn a subprocess in Python have tried to map almost directly to the underlying C API... which is pretty awful.
> For some reason, all of the ways to spawn a subprocess
> in Python have tried to map almost directly to the
> underlying C API
In the early days python had a principle that there should be one way to do things. You don't hear this so much any more: we're long past that now, with some things different between 2.6 and 2.7 (arg handling), and with multiple broken libraries in the stdlib. When you're working on your own computer and your own time with root access you can always hand-roll outcmes. But it's common to have to deal with a spread of python's and cater to the most obsolete version. Yet I suspect some people still aspire to the one-way-to-do-it, and pretend it's true.
I think we should dump the principle.
A good example of why compromise is not the right outcome is the curses library - it's not quite Stevens, but it's not friendly either. It's hard to do good work with the curses library. We'd be better off if there was (1) a close curses mapping to the C ncurses mechanisms and (2) a nice-to-use abstraction layer that hid far more away from you.
For quick tasks and scripts, I've found subprocess.check_call, and subprocess.check_output with shell=True are great tools for spawning subprocesses and quickly grabbing output. They're pretty straightforward to use.
There are lots of interesting corner cases, for example how to join stdout and stderr properly without blocking on one stream while the other is overflowing.
On the other hand, almost nobody ever needs this. Ruby's "output = `command`" probably covers 90% of the use cases with the most trivial API imaginable. The hard part obviously is exposing the advanced functionality without compromising on the simplicity.
Almost all programming communities can learn a lot from Ruby's "if it's too hard, you're not cheating enough" approach (dhh quote I believe). Yes, the process could return an exabyte of stdout data, but do you really care? Is that really the problem this API should try to solve, with all special cases? That's not good computer science practice, but surprisingly effective.
Using made up stats, slurping the entire output of a process in a big string would fail 99% of the times 30 years ago, 50% of the times 20 years ago, 1% of the times 10 years ago, but less than 0.01% of the time now. You'd waste your time doing it 'the right way'.
So it is with most simple data processing. If your goal is just to ship a product fast, you no longer need the old type of smart programmers; nowadays, smart means doing it fast and badly.
Here's a way to do it with Twisted (docs here: http://twistedmatrix.com/documents/current/core/howto/proces... ):
from twisted.internet import reactor, protocol
class PrintAndLogProtocol(protocol.ProcessProtocol):
def outReceived(self, data):
# print and log
errReceived = outReceived
reactor.spawnProcess(PrintAndLogProtocol(),
'/path/to/exe', ['exe', 'arg1', 'arg2'])
reactor.run() import os
import sys
import time
import subprocess
# turn off stdout buffering. otherwise we won't see things like wget progress-bars that update without newlines.
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
pipename = "tempfile"
if os.path.exists(pipename):
os.remove(pipename)
# create a pipe. one side is connected to the ping process, other side is connected to python.
os.mkfifo(pipename)
read_fd = os.open(pipename, os.O_RDONLY|os.O_NONBLOCK)
writer = open(pipename, "w+")
proc = subprocess.Popen("ping www.google.com", cwd=sys.path[0], stdout=writer, stderr=writer, shell=True)
while 1:
try:
# nonblocking poll data from the external process.
s = os.read(read_fd, 1024)
if s:
sys.stdout.write(s)
except OSError:
pass
# sidenote: minimum sleep time is 1/64 seconds on many windows pc-s.
time.sleep(0.1)
# remember to remove the pipe "tempfile" import os, sys, time, subprocess, fcntl
sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)
read_fd, write_fd = os.pipe()
fcntl.fcntl(read_fd, fcntl.F_SETFL, os.O_NONBLOCK) # don't know of any windows equivalent for this line
proc = subprocess.Popen("ping www.google.com", cwd=sys.path[0], stdout=write_fd, stderr=write_fd, shell=True)
while 1:
try:
s = os.read(read_fd, 1024)
if s:
sys.stdout.write(s)
except OSError:
pass
time.sleep(0.1)After that use virtualenv with virtualenvwrapper.
I guess you really stick by that rule of "one way to do it", huh? I kid. Great presentation, I like your passion and I hope that your call to arms is answered.
Also, it breaks the back button.
Esp. the "installing python" one. Just use your package manager to install all the versions you need.
And for "Packaging and Dependencies", just use pip.
I know it was annoying how long RHEL lasted with Python 2.4, but now the latest RHEL, Debian-stable, and Ubuntu LTS all have Python 2.6, so write to Python 2.6 if your company uses those OSs, there's not excuse to write incompatible code targeted at Python 2.7.
In many Linux distributions it is impossible to replace the Python version, core parts of the OS like package management will break. It is possible to install another version in parallel and use "virtualenv --python=" to use it for an application. If you are using RHEL/CentOS 5 you can install the python26 package from the EPEL repository.
Mostly I agree with you, but installing python on windows is a PITA compared to *nix.
PIP works until you need something that requires compiling extensions which is also a PITA on windows.
IIRC easy_install can install binaries, but I could be wrong about that since I don't do windows development anymore and thus pip works for me 99% of the time.
Regardless, my point is, though I think we all wish it was as simple as using your package manager and pip, it simply isn't that way for everyone.
And it probably never will be, but we can always make it simpler. :)
Whether you or I prefer it or not, there are lots of people who either by choice or not develop on windows and in some cases deploy to windows servers.
And for those people, things aren't as simple as "use the package manager and pip" no matter how much anyone would like it to be.
reality sucks.
You could use *.exe installers such as http://www.lfd.uci.edu/~gohlke/pythonlibs/
Advanced users could use mingw to compile extensions.
(It is, however, poorly documented.)
And installing packages into the system python (if that's what you're suggesting) is the path to madness. It's much better to use virtualenvs you can throw away at will. All in all, it's usually best just to leave the system Python alone to avoid causing problems with any other packages that may depend on it being in a consistent state.
Unlike startups, we enterprise dudes don't have the liberty of choosing the latest distros with better base Python versions.in some cases,the machines don't even have a compiler installed.
Try running your funky new admin script on these rhel5 boxen is a pain.
I've resorted to compiling with LD_RUN_PATH, -rpath & -r set.
I think this should be included in the python-guide for the sanity of other devops dudes
The standard library needs a reboot. Why not do it in Python 3? Nobody's using it yet anyway ;-)
Still I like these new projects; it's a shame they missed the python 3.x boat by only a year or two. That would have been a great time to include them in the stdlib.
For example, the standard library already has urllib2, which provides HTTP support much better than most other languages' standard libraries. But Requests is a rewrite of it, taking ease of use to a whole new level without compromising any of the features. There is no equivalent in any other language.
I don't know, LWP is pretty easy to use.
IMO, Python's stdlib beats C++, Ruby, Java, javascript and Perl....
Vast amounts of it are actually incomplete or outright broken when you try to use it in practice.
The reliable parts of Python's standard library are at a level little beyond C and C++ and far behind Perl/CPAN, Java, .Net and so on.
And even then, as the article we're discussing points out, the APIs aren't actually very good a lot of the time.
As someone who has spent the last 5.5 years at a job where I didn't need anything but the standard library, I don't agree. What are these other languages that contain a better situation?
If you say a "std lib" is something that is installed along with the language, then CPAN fits the bill.
$ corelist CPAN
CPAN was first released with perl 5.004Even if CPAN was a standard library, that which you install from it is clearly not, which is what the argument to date has centered around.