So Pythonistas, you want to get rid of the GIL... | Hacker News Reader

So Pythonistas, you want to get rid of the GIL...

functional-orbitz.blogspot.se

34 pointssausagefeet29 comments

> You will rewrite your code to not make use of the guarantees.

I suppose a problem is that unlike Java, Python is not fast enough to implement low level data structures in the language itself, so you can't easily implement non-thread-safe data structures without losing memory safety at the C level, which is unacceptable.

Still, I have the feeling that with heavy use of biased locking, you could stick locks everywhere without a lot of overhead in the non-contended case. Or maybe that famous patch just needs to be revisited, tweaked to more efficiently address the areas of highest contention... see http://dabeaz.blogspot.com/2011/08/inside-look-at-gil-remova... for some context.

(some forms of biased locking are patented.)

Python objects only need to be thread-safe if they are accessed by multiple threads. A proven solution that could preserve Python's atomicity guarantees is actor-based concurrency like Erlang or Go. When Python objects are queued to an actor's channel, a deep copy (or copy-on-write reference) could be sent to avoid data sharing.

The blog's example is stupid. Of course if you pop a list in more than one thread the behavior is undefined. Who on earth doesn't expect that?

@jagw: ok fair, but that is going to be the very first example of things thou shalt not do when using threads in any article, book, or course

People who have never done proper multi-threaded programming.

It's defined in Python.

Do actual pythonistas really hate the GIL? I thought it was mostly a troll talking point, just like the whitespace.

Why would the inability to write multi-threaded programs that are worth a damn be a troll talking point? It meaningfully lowers the appropriateness of using python for server code.

There's more than one way to do it. If you're thinking about hundreds of threads doing chunks of work, then my comment will not be meaningful but if you're talking about maximizing CPU utilization for a particular task, then multiple processes is a valid and proven approach (Tornado does just that for example, forking as many processes as you have CPUs and putting them to work).

The viability of separate processes is highly dependent on the problem domain and algorithm. Threads have their place.

No, the GIL is simply not relevant for real work. True Pythonistas, when performance is critical, just drop into C for the crucial section and use Python to control it. When C code is running, the Python runtime isn't blocked - you can kick off a hundred native pthreads in the background if you want to.

This is also true in OCaml BTW, but it's trickier to make the call as when running single-threaded, OCaml code is as fast as C anyway. A Python/OCaml hybrid using C as glue is probably the best of both worlds.

or you look at multiprocessing, which spreads the computation into multiple processes, while enforcing clear boundaries and communication methods.

The main problem is the perception that code would linearly scale if the GIL was removed. Eg on a 4 core machine code would run 4 times as fast. And that the GIL has no value. Both statements are completely false. (A bigger issue is the reference counting anyway. Changing that should be a precursor to GIL removal.)

I'm an author of a C extension (apsw) and the GIL is fantastic. Without it the code would be considerably more complicated, longer and likely to have race conditions and similar gnarly bugs.

As a Python developer I use:

* Multiple processes usually by executing a new process which allows me to easily substitute testing processes and using stdin/stdout * Use networking for talking between parts (allows spreading over multiple machines, failover, redundancy, load balancing etc) * multiprocessing for pure Python work * Threads for I/O bound work or work that waits on other things * C extensions that release the GIL

Actually getting rid of the GIL wouldn't help.

I'm actually interested to know what the HN community does to achieve parallelism in Python? Do you use multiprocessing module, Stackless python, PP module (parallel python)?

Either multiprocessing or shifting the bits that really benefit from parallelization out to C/C++ tend to cover the vast majority of my needs. Also numpy and numexpr does a lot of the multiprocessing stuff for me, so in many cases I don't even have to think about it.

That's usually the point at which I decide the project could really use being rewritten in C.

If I need performance badly enough that I'm considering making major structural changes in order to utilise multi-process or find some other way around the GIL, I probably ought to be considering using a language that supports what I'm trying to do. Of course my python projects tend to be fairly small.

Or just write the fast bits in C, call it from Python via ctypes and get the GIL unlocked and parallelism for free.

If my projects were large enough to make that a more attractive option, sure, that's a great plan.

Usually when I've hit this situation I'm writing something simple but processor intensive, and it's easier for me (12 years commercial parallel/distributed C programming, amateur python enthusiast) to go back to home ground at this point.

For sufficiently new version of Python (3.2+), concurrent.futures[1] is pretty nice for its use-case, but otherwise multiprocessing has been quite good.

[1]: http://docs.python.org/py3k/library/concurrent.futures.html

If the problem at hand isn't easily solved with a multi-process approach then this would probably be the time I'd start taking a hard look at another language.

Scala, Go, Clojure etc make this so much easier that I think a language jump is reasonable.

I use workers (sometimes multiprocessing, sometimes I do my own starting of them).

Often I have use-cases where my own shared memory system works well http://williamedwardscoder.tumblr.com/post/13363076806/buffc...

> I was writing a custom video editing program in Python (interesting choice of language for that problem) and I wanted to cache decoded frames

I'm using QThread for PyQt GUI applications. However the GIL makes it impossible to execute _Python_ code in parallel. I can understand the reasons for that, however it doesn't change the fact that it makes my application inherently slower and is therefore something I dislike.

Apart from that I really like Python.

Same here. As a nice side effect, QThread somehow makes signals work again, so you can Ctrl-C a MT Python program (you still have to set a handler for SIGINT but at least it works).

For this reason, I actually use QThread in console apps aswell.

If you are using QThread for console apps and are not using any other Qt libraries, isn't your console app has an unnecessary dependency on Qt ?

Qt's split up into separate libraries; QtCore is pretty much a general portable utility library like glib or APR, and it's perfectly reasonable for a console app to depend on it.

Well I guess it is "unnecessary" to the extent it is unnecessary to be able to Control-C your programs. Personally I like my programs to respond to signals.

multiprocessing or rewrite in something else, depending on the nature of the problem.

Could we not have a per-object lock, but only on objects the programmer defines as requiring thread safety?

If you tag objects as being either "owned" by a thread or "shared" then you could (in the python implementation) do something like:

  if (obj.owner == current_thread):
    # just get in and mess with it
  elsif (obj.owner == shared):
    # get in and mess with it under a lock
  else
    throw "not your obj - make this obj shared if you need that"

From a language perspective, you'd just need a way to declare that an object was 'shared'. All other objs would be owned by their creating thread.

No, that doesn't work. Java has something like this (synchronized methods) but they are cumbersome (I find programming nicer with plain old locks and conditions, the Java synchronized model feels just a layer of cruft but its all lock&cond inside).

But Python is different than Java. The Python interpreter has to guard the internal data structures of the interpreter itself, and that is a lot more tricky than it seems on the surface.

Reply on news.ycombinator.com