But if you're writing to the same object from two different threads you're going to have undefined behavior regardless of the GIL, yes?
So with Python concurrency, you can get unpredictable behavior (such as two threads losing values when incrementing a counter), but not undefined behavior in the C sense, such as use-after-free.
The GIL exists to protect the interpreters internal data, not your applications data. If you access mutable data from more than one thread, you still need to your own synchronisation.
One example would be incrementing a counter for statistics purposes. If the counter is atomic, and the reader of the value is ok with a slightly out of date value, it's fine. If code is doing this in GIL Python, it's working now, and will break after the GIL is removed.
Atomics are isolated to where they are needed, the GIL is global though C (but not Python) code can release it.
https://stackoverflow.com/a/1717514
The Python documentation seems misleading to me on this:
> In theory, this means an exact accounting requires an exact understanding of the PVM bytecode implementation. In practice, it means that operations on shared variables of built-in data types (ints, lists, dicts, etc) that “look atomic” really are.
count = 0
def inc():
count += 1
On the other hand, I think you could build a horribly inefficient actual atomic counter with
count = []
def inc():
count.append(None)
def get_count():
return len(count)
> A lot of work has gone into the list and dict implementations to make them thread-safe. And so on.
So removing the GIL might not be a problem.
Regarding list, it sounds like it might actually keep working atomically without the GIL:
> A lot of work has gone into the list and dict implementations to make them thread-safe. And so on.
if
I know you came to the same conclusion in another comments, but here's a look at it using the `dis` module:
a += 1
LOAD_FAST 1 (loads b)
LOAD_CONST 1 (loads the constant 1)
INPLACE_ADD (perform the addition)
STORE_FAST 1 (store back into b)
From your other comment:
> so even though an increment on a loaded int is atomic, loading it and storing it aren't
Its always the loads and stores that are the problem.
But that's the problem with relying on the GIL: it has lock in the name, but it does not protect you, unless you understand the internals and know what you're doing. It protects the interpreter. This isn't much different from programming in other languages without a GIL: if you understand the internals what you're doing you may or may not need locks, because you will know what is and isn't atomic (and even when things are atomic, its still difficult to write thread-safe code! lock-free algorithms are much harder than using mutexes).
Thread safe code requires thinking hard about your code, the GIL does not protect you from that.
For instance, in your example, I know that the call to the C code to do the list operations is atomic (a single bytecode instruction), but I can't assume that all such calls to C code are safe because of this, unless I know for sure that the C code doesn't itself release the GIL. I assume that simple calls like list append/pop wouldn't have any reason to do this, but I can't assume this for any given function/method call that delegates to C, since some calls do release the GIL.
So, with or without GIL, you either really need to understand what's going on under the hood so you can avoid using locks in your code (GIL or atomics-based lock-free programming), or you use locks (mutex, semaphore, condition variables etc). No matter what you do, to write thread safe programs, you need to understand what you're doing and how things are synchronizing and operating. The GIL doesn't remove that need.
Of course, removing the GIL removes one possible implementation option, I just don't believe the GIL really makes it any easier. Once you know enough internals to know what is and isn't safe with the GIL, you could just as easily do your own syncrhonization.