Hacker News Reader

Comment by amirmansour | Hacker News Reader

I'm actually interested to know what the HN community does to achieve parallelism in Python? Do you use multiprocessing module, Stackless python, PP module (parallel python)?

View on HN

Either multiprocessing or shifting the bits that really benefit from parallelization out to C/C++ tend to cover the vast majority of my needs. Also numpy and numexpr does a lot of the multiprocessing stuff for me, so in many cases I don't even have to think about it.

That's usually the point at which I decide the project could really use being rewritten in C.

If I need performance badly enough that I'm considering making major structural changes in order to utilise multi-process or find some other way around the GIL, I probably ought to be considering using a language that supports what I'm trying to do. Of course my python projects tend to be fairly small.

Or just write the fast bits in C, call it from Python via ctypes and get the GIL unlocked and parallelism for free.

If my projects were large enough to make that a more attractive option, sure, that's a great plan.

Usually when I've hit this situation I'm writing something simple but processor intensive, and it's easier for me (12 years commercial parallel/distributed C programming, amateur python enthusiast) to go back to home ground at this point.

For sufficiently new version of Python (3.2+), concurrent.futures[1] is pretty nice for its use-case, but otherwise multiprocessing has been quite good.

[1]: http://docs.python.org/py3k/library/concurrent.futures.html

If the problem at hand isn't easily solved with a multi-process approach then this would probably be the time I'd start taking a hard look at another language.

Scala, Go, Clojure etc make this so much easier that I think a language jump is reasonable.

I use workers (sometimes multiprocessing, sometimes I do my own starting of them).

Often I have use-cases where my own shared memory system works well http://williamedwardscoder.tumblr.com/post/13363076806/buffc...

> I was writing a custom video editing program in Python (interesting choice of language for that problem) and I wanted to cache decoded frames

I'm using QThread for PyQt GUI applications. However the GIL makes it impossible to execute _Python_ code in parallel. I can understand the reasons for that, however it doesn't change the fact that it makes my application inherently slower and is therefore something I dislike.

Apart from that I really like Python.

Same here. As a nice side effect, QThread somehow makes signals work again, so you can Ctrl-C a MT Python program (you still have to set a handler for SIGINT but at least it works).

For this reason, I actually use QThread in console apps aswell.

If you are using QThread for console apps and are not using any other Qt libraries, isn't your console app has an unnecessary dependency on Qt ?

Well I guess it is "unnecessary" to the extent it is unnecessary to be able to Control-C your programs. Personally I like my programs to respond to signals.

Qt's split up into separate libraries; QtCore is pretty much a general portable utility library like glib or APR, and it's perfectly reasonable for a console app to depend on it.

multiprocessing or rewrite in something else, depending on the nature of the problem.

Reply on news.ycombinator.com