This is _very_ different to the 2-to-3 transition where absolutely every single person, even those who couldn't care less, had to change their code if they wanted to use python 3.
This is _very_ different to the 2-to-3 transition where absolutely every single person, even those who couldn't care less, had to change their code if they wanted to use python 3.
Oh, so naive... All the mutation code in Python which "worked" because Python didn't really have any real concurrency. Add to it -- there's no real plan about what to do with Python concurrency. Removing GIL is only one "half" of the problem, you need to give developers some sort of a framework to use to deal with concurrency. Python's threads are extremely underdeveloped and dangerous to use. Python doesn't even have anything like "synchronized" from the Java world. So, all synchronization requires dealing with locks, mutexes, condition variables...
Most Python programs today didn't bother to deal with threads because they didn't confer enough benefits to be worth using. So, "automatically" parallelizing Python code, as in allowing it to run in actual threads is going to bring about lots and lots of bugs in trivial code written by people with no clue about concurrency.
As always, by far the best way to interact between threads is to use thread-safe queues (AKA message passing). Luckily, Python has one of those [1]. No complicated synchronisation needed.
The reason to want threads is to be able to share memory. That's literally why they were created. If you are sending messages instead of sharing memory, you don't need threads. You need something like Erlang processes.
The problem is that people who wrote Python never had a plan. They sucked and still suck as programmers. So... they knew there are threads. And it was easy to write a bunch of wrappers around pthreads. And that's what they did. And then they realized they don't know how to deal with concurrency, so they found a simple way out -- GIL.
The whole history of Python is the history of choosing the easy but wrong. And it's probably the only consistent thing about the language.
Yes you could use multiple processes, but you have the extra expense of serialisation, or you could use shared memory but you'd have to administer that and you'd still have the expense of context switches. And inevitably, yes, there is some actual shared state like a logger and modules that you've loaded, which again would be a pain in multiple processes.
You can call a Python thread plus a queue an Erlang process if you like, or say that I should use Erlang processes instead. But the fact is, the Python version works perfectly well for many problems. It does all the things that you typically need if threads: shares state (via the queues), let's you concurrently use the CPU (via C libraries that release the GIL – but no GIL would be even better), and writing blocking IO if you wish. Not missing the point at all.
The developers of Python didn't "suck as programmers" and it doesn't help your point to claim they do. Guido choose to use the GIL because he was OK with multiple threads but not at the expense of single thread performance, and no one showed any solution to that that beats the GIL – until now. (Personally I think the trade off was wrong, and a small hit to single threaded performance would have been with it. But that's different from being ignorant to the fact there was an actual reason.)
[1] https://docs.python.org/3/library/asyncio-eventloop.html#asy...
It's a thing completely separate from threads, where no code is supposed to run concurrently. The idea of this feature is that a program may schedule a bunch of I/O operations and then wait for their completion instead of scheduling I/O operations one at a time.
As of now, this is an obsolete mechanism of dealing with I/O as now we have uring_io. But, truth be told, it never really worked well... I mean, if you knew what you were doing, you could have taken advantage of this feature, but it was never in shape to be library-grade multi-purpose functionality.
Python made a stupid bet on it and encoded it into the language through async / await keywords. But if this was the only stupid thing Python has done in its history, flying cars and hoverboards would probably be an integral part of our daily lives.
Async/await is concurrent, that’s the whole point. Its not usually parallel, because the asyncio runtime (and, IIRC, all the major alternate runtimes) schedules tasks on the same thread, and if there was a multithreaded runtime, its parallelism would be limited by the GIL to only actually having multiple threads making progress if all but one were in native code that released the GIL.
> nearly every other language that has async/await (or promises) is concurrent-by-default.
JavaScript isn't parallel for async/await. Ruby has multiple async/promises implementations, some of which are parallel (use separate threads) to some degree even with the GVL (which is like Python’s GIL), and others are not. (all are, of course, concurrent.)
The GIL limits the value of a multithreading async/await runtime, but it doesn’t prevent it, and a GILectomy doesn’t buy you one for free (or make a multithreading a cost-free choice.)
Not true at all. Plenty of people (including me) use threads in Python for:
* Blocking I/O
* CPU heavy libraries written in C (as those release the GIL)
They work fine, even with the GIL. They only work badly if you want to run a lot of pure-Python (non-I/O) code in multiple threads - which, fair enough, sometimes you might want to do, and the GIL is a problem for that.
Not at all. I'm saying that a sizable portion of Python libraries is completely unaware of threads. But they can still take foreign-own object and operate on them as if threads didn't exist.
So, imagine a simplified hypothetical scenario, where one library has a function for counting keys in a dictionary. This library was written by someone unaware and unwilling to acknowledge thread existence. So, if the dictionary it counts the keys of is modified in a separate thread -- boom! But, third-party code using that library has no easy way of knowing if the library is prepared to deal with threads, and may have been using it for a while, until, again boom!
Now, to make this more concrete: have you ever heard of Boto3, the AWS client library? Well, it does roughly what's described in the paragraph above -- it manipulates a bunch of its own objects in a non-thread-safe way. But, you would really want to use it in threads because that makes it so much easier to manage things like rate-limiting (across multiple clients), and, obviously, you don't want to deploy a large fleet of VMs one-by-one. The end result? -- boom!
If you assume nogil, you need to choose dependencies that support that. You may need to trade off: eschew dependencies that aren't looking like they will be nogil compatible by the deadline. You are stuck on Python 3.18 maintenance branch or whatever, rather than the 3.19 (in reality .. 4.0) version.
Or choose gil then you can use everything. But is there a prisoners dilemma - everyone picks gil, uses whatever dependencies, library maintainers assuming this don't bother to add nogil support, and then the decision becomes to stick to gil, which if you suspect will happen makes you reason even harder not to support nogil.
Because of this I don’t expect there to be multiple versions of any library. Once a library does the (admittedly heavy) lift to no GIL it will just be the main version of that library going forward.
Then on top of that, they know their effort might be for nothing if the decision is made to keep Python GIL-only all along (one of the possible 3 outcomes at the end of the 5 years: ["gil", "nogil", "both supported").
I'm wondering how many libraries with binary extensions are actually in common use. Like, maybe 90% of python projects use a subset of a few hundred such packages?
That's a hassle if you maintain one of those packages, and will be a bit disappointing if in 5 years' time you're still depending on GIL-reliant packages.
But it's nothing like the chaos of the python 2-3 changes, where ~100% of python files in every package and end-user project had to be fixed.
I only learned about this this morning though, it's very possible I'm missing something. A lot of the concerns people are raising look a bit overblown to me.
I take the point that after so many abortive GIL removal attempts, it's harder to be confident this one will happen. But having the go-ahead from the steering council seems like a good indicator this one has traction.
You could have a single new ABI supporting both no-GIL and with-GIL, but it wouldn't be compatible with the existing stable ABI.
And that's talking about multithreaded code. I bet that even with noGIL, lots of Python code will still continue to be single-threaded, making the gil/no-gil decision irrelevant (save for those native extensions).