If PEP 703 is accepted, Meta can commit three engineer-years to no-GIL CPython
discuss.python.org
discuss.python.org
https://peps.python.org/pep-0703/#backwards-compatibility
That section is short and I encourage you to read it in full. Here's what it says about how removing the GIL affects existing Python code:
• Destructors and weak reference callbacks for code objects and top-level function objects are delayed until the next cyclic garbage collection due to the use of deferred reference counting.
• Destructors for some objects accessed by multiple threads may be delayed slightly due to biased reference counting. This is rare: most objects, even those accessed by multiple threads, are destroyed immediately as soon as their reference counts are zero. Two places in the Python standard library tests required gc.collect() calls to continue to pass.
That's it with respect to Python code.
Removing the GIL will require a new ABI, so existing C-API extensions will minimally need to be rebuilt and may also require other changes. Updating C-API extensions is where the majority of work will be if the PEP is accepted.
This will be an opt-in feature by building the interpreter using `--disable-gil`. The PEP is currently targeted at Python 3.13.
This is nothing like the Python 2 > 3 transition.
which python3
which python3nogil
Python4 needs a hard reset.
venv == virtualenv
virtualenvwrapper is ancient not rly used anymore
pyenv is a third party tool that makes some of this easier, notably around creating more than just a virtual env in that you also choose the Python version.
Python is not hard to deal with in this regard I think people are just uninformed.
The literal only thing you need to understand is “sys.path”. If you inspect this in a shell you will know what you’re up against. Python is all literally just directories. It’s so easy and yet people get so bent out of shape over it.
Create a venv, activate it, and use pip as normal. If you ever run into issues, look at sys.path. That’s it.
And few things have changed since then.
And you can disagree all you want, but it's simply wrong that Python's packaging/venv ecosystem is "just fine".
Is it because people have to use venvs that people complain about it?
I’ll admit being able to install via an OS package manager, vs pip, vs anaconda etc etc can be confusing, but is any of that really Python (the language)’s fault?
pip install
pip install -u
sudo pip install
conda install
sudo conda install
some packages require one, are fine with other with few small warnings, and dont work with third way of installing.
I don’t know the -u flag on pip, never used it can’t find it in the docs.
With a virtual environment sudo is not needed. Assuming you created it, and/or it is owned by you.
Virtual environments are just directories on disk. They are not complex.
I don’t use conda because it’s never felt even remotely necessary to me.
pip and a requirements file is all you need.
how about when you are authoring script under your name, but then want to schedule it for cron to run periodically?
I often find myself working under my user on remote server, but then I want to schedule cron job - and run into all sorts of permissions / bugs and missing packages.
especially when multiple machines need to run this script, and I don't want to involve containers to run 20-lines simple python script.
this is why Golang is so popular - you can just scp a single binary across machines and it will just work.
1. pip install (with/out --user flag )
2. pip3 install (with/out --user flag )
3. sudo pip install
4. sudo pip3 install
5. conda install
6. sudo conda install
sudo apt install python3ispython
sudo update-alternatives --set python3 /usr/bin/python3-nogil
sudo update-alternatives --set python /usr/bin/python3Having interpreters and packages strewn across the machine is a nightmare. The lack of standard tooling has created a lawlessly dangerous wild west. There are no maps, no guardrails, and you have to beware of the hidden snakes. It goes against the zen of python.
As a counter example, Rust packs everything in hermetically from the start. Python4 [1] could use this as inspiration. Cargo is what package and version management should be, and other languages should adopt its lessons.
[1] Let's make a clean break from Python3 even if we don't need a new version right now.
Plus, you now have to teach and evangelize your method versus the dozens of others out there. It's crazy town.
The negative thoughts and feelings I once had for PHP are now directed mostly at Python. PHP fixed a lot of its problems over the last decade. Python has picked up considerable baggage in that time. It needs to take the time to do the same cleanup and standardization.
I was describing a workflow that works for me to someone who didn't seem to have found an effective Python workflow in hopes that it can work for them too. I work across a variety of languages and none that I've worked with doesn't have some issue that I can't complain about[1]. I personally don't find Python all that painful to work with (and I've been working with it since 1.5.2), but I understand my experience is not universal.
[1] If it's not the language, it's the dependency manager. If it's not the dependency manager, it's the error handing. If it's not the error handling, it's the build process. If it's not the build process, it's the community. If not the community, the tooling. Etc. I have some languages I like more and some less. Mostly it comes down to taste. I'm not here to apologize for or defend Python. I'm only here to describe how I use it effectively, and to correct what I thought were inaccuracies with respect to removing the GIL.
For this you dont neet .envrc and direnv, as this is handled perfectly fine by peen itself: pyenv local <pyenv / virtualenv name>
https://github.com/pyenv/pyenv
I use direnv because I work with many languages and repos and I don't want each language's version manager linked into my shell's profile. As well, direnv lets me control things besides the language version. Finally, direnv means I don't have to explicitly run any commands to set things up. I just cd to a directory.
Its insane that nor Node nor Python have a first class version selector.
I don't think it's true. rustup installs new version only when you run `rustup update`. What parent is talking about is pinning a particular rustc version in Cargo.toml, which allows rustup to download that version of rustc to build that particular project/crate.
Consider R, which is filled with the same kind of people. There's one package repository and if your package doesn't build cleanly under the latest version of R, it's removed from the repo.
Don't get me wrong, this has other problems but at least it means that all packages will work with a single language version.
That being said, after two decades of using Python professionally, the only really problems I’ve ever encountered are “package doesn’t support this version for {reasons}” and “ML library is doing something undocumented and/or dumb that requires a specific Python version.” The former is normally because the package author is no longer maintaining their package and the latter is because, again, the ML community is among the absolute worst at creating solid tooling.
However, Python (1991) is only 2 yrs older than R (1993)
That's a vast exaggeration. It is not "really really" hard to spin up a venv and specify your requirements. People just don't do it, and blame the tools for what are bad engineering practices agnostic to any language.
"Really really" not easy would be handling C, C++, etc. dependencies.
I really really wish it was, as then I wouldn't have had to learn Docker.
Maybe more time than getting python deps to work but more deterministic and takes less cleverness.
Similar problem could be with changing behavior for divisions (which actually was more challenging) but similarly you could enable that behavior.
The main problem with migration though was addition of Unicode. You can't just enable it on file by file basis, because once you enable the new behavior in a single file you, will start passing arguments in Unicode to other code in other files and if that code wasn't adapted or will break.
And it was even worse than that because that problem extended to your dependencies as well. Ideally dependencies should be updated first, then your application, but since python 2 was still supported (for a decade after python 3 was released) then there was no motivation to do it.
And if that wasn't enough python 2 already had Unicode support added, but that implementation was incorrect, so even if you imported Unicode_literals from __future__ you potentially broke compatibility with existing python 2 dependencies without guarantee that your code will work on python 3.
IMO that particular change couldn't be done with pragmas, the core issue is that python 3 put a clear separation between text and binary data, but Python 2 mangled them together. That still was true even when you used Unicode in python 2.
The proper way to perform the migration IMO would be to type annotate the code. And then run mypy check in python 3 mode.
But during the time it took to design and deliver Python 3, the language exploded in popularity and reached a much wider audience and 3rd party libraries like numpy became crucial. So when Python 3 was ready it was a completely different ecosystem which was much harder to migrate. But I dont think the core team really relized that before it was too late.
/home/you/project/venv/bin/python
As for managing Python library dependencies, I use poetry (https://python-poetry.org), though unfortunately both it and pipenv seem to progressively break functionality over time for some reason.
Because the code that is updated is expected to work with both versions.
> The --disable-gil builds of CPython will still support optionally running with the GIL enabled at runtime (see PYTHONGIL Environment Variable and Py_mod_gil Slot).
Well what py2to3 promised was also not what happened and by far one of worst disasters of a migration in history of open source so you can't blame people for being skeptical
"It didn't happen"
Only difference is that they noticed it's a bad idea midway and went back to developing Perl 5
And I still can write "use v5.8" in my Perl script and write script that works in Centos 5 instance from 2007, if I needed to do something on legacy system, and it will work the same on latest Perl.
> because at the time Perl was more popular than Python.
I don't believe that was true at the time as P6 took like decade to decide on semantics and only then the implementations started to happen. More existing code, sure, but much of the web dev moved to PHP/Ruby and many other things to Python.
And unlike Py3 it was initially also much slower than P5, making migration kinda worthless.
I'm not saying for sure this is bad! I do think it is dishonest though about the potential impact. Lots of critical libraries are written in C.
This was the top comment when I added my original comment:
"Fundamentally every single piece of python code ever written will have to stop and now worry about potential race conditions."
https://news.ycombinator.com/item?id=36644114
Most of the other comments here at the time were similarly from people who clearly hadn't read the PEP saying it would break all existing Python code.
So I did my best to represent the backwards compatibility section of the PEP. I told folks to go read it and linked to it. I cited the portion relevant to Python code. There were too many bullet points for the C-API, so I summarized it with the disclaimer "Updating C-API extensions is where the majority of work will be if the PEP is accepted."
I also read the PEP discussion thread where there was disagreement from Python maintainers on how much work would be required of C extension authors, but most of the folks stating it would be a lot of work hadn't seemed to actually have tried to port anything to the new API. Meanwhile Sam had asserted that:
> Most C API extensions don’t require any changes, and for those that do require changes, the changes are small. For example, for “nogil” Python I’m providing binary wheels for ~35 extensions that are slow or difficult to build from source and only about seven projects required code changes (PyTorch, pybind11, Cython, numpy, scikit-learn, viztracer, pyo3). For four of those projects, the code changes have already been contributed and adopted upstream. For comparison, many of those same projects also frequently require changes to support minor releases of CPython.
I don't think I've misrepresented anything.
> No-GIL ain't done until all the C extensions run, especially NumPy.
Sam already got NumPy working without the gil:
* C function called with GIL held
* Extract data needed to do work
* Release GIL
* Do work
* Reacquire GIL
* Modify data, build result
* Return
As an example of the changes, a list could be passed in. I would need some form of locking while processing that list so that mutations while processing won't crash the code.
The GIL does currently result in robust code by default because data can't mutate underneath you. Without the GIL the code will appear to work, but it will be trivial for an attacker to use mutations to crash the code. Expect huge numbers of CVEs.
This is the lead author of the majorly potential impacted extension!
Borrowed references can be more problematic, but it seems most cases could be fixed by replacing GetItem with FetchItem calls.
Overall, as another C Python extension, I don’t really think it’s going to be that much of a pain. In fact, I could even get away with no changes (other than build fixes and such) if, for example, I guarantee that each “main object instance” is only accessed by one thread, and I’d still get a lot of benefits from the nogil.
if (PyList_Size(lst) >= 3) {
PyList_SetItem(lst, 0, ...);
PyList_SetItem(lst, 1, ...);
PyList_SetItem(lst, 2, ...);
}
IOW, your example is not much different than the equivalent Python code:
if len(lst) >= 3:
lst[0] = "..."
lst[1] = "..."
lst[2] = "..."
Let's look at the interpreter code:
1. PyList_SetItem calls Py_XSETREF(dst, src) where dst is the list position:
https://github.com/python/cpython/blob/ee46cb6aa959d891b0a48...
2. Py_XSETREF calls Py_XDECREF on the old item:
https://github.com/python/cpython/blob/ca8b55c7f54b38e264056...
3. Py_XDECREF calls Py_DECREF on non-Null objects:
https://github.com/python/cpython/blob/main/Include/object.h...
4. Py_DECREF can cause arbitrary code to run:
Warning: The deallocation function can cause arbitrary Python code to be invoked (e.g. when a class instance with a __del__() method is deallocated).
https://docs.python.org/3/c-api/refcounting.html#c.Py_DECREF
This is backed up by the documentation for PyList_SetItem:
This function “steals” a reference to item and discards a reference to an item already in the list at the affected position.
https://docs.python.org/3.8/c-api/list.html#c.PyList_SetItem
The "Defining Extension Types: Tutorial" also includes this warning in an example that uses Py_XDECREF:
But this would be risky. Our type doesn’t restrict the type of the first member, so it could be any kind of object. It could have a destructor that causes code to be executed that tries to access the first member; or that destructor could release the Global interpreter Lock and let arbitrary code run in other threads that accesses and modifies our object.
https://docs.python.org/3/extending/newtypes_tutorial.html#a...
I dunno, I'd rather have a reliable bug than a heisenbug.
Any class with a __del__() method is going to release the GIL. That's not uncommon. My intuition is that threaded Python is rarer than classes with a __del__() method.
> Most C API extensions don’t require any changes, and for those that do require changes, the changes are small. For example, for “nogil” Python I’m providing binary wheels for ~35 extensions that are slow or difficult to build from source and only about seven projects required code changes (PyTorch, pybind11, Cython, numpy, scikit-learn, viztracer, pyo3). For four of those projects, the code changes have already been contributed and adopted upstream. For comparison, many of those same projects also frequently require changes to support minor releases of CPython.
If you're using the GIL to protect access to non-Python objects, that will need to change.
The PEP mentions a future HOWTO on how to updating existing extensions. I wish that were already written.
There's disagreement among Python maintainers in the discussion thread on the PEP in how much work will be involved and I don't expect to resolve it here.
This is automatic via the PyList_GetItem/SetItem API, so I guess the error you're talking about is that you read a list y = [A, ...], your code reads A and copies the data to (new) A2, and then you iterate over it again and see that A != A2 because another thread has modified y?
This sounds like a nightmare waiting to happen.
this actually does "break" a lot of things, as you would be surprised how much code implicitly relies upon cPython's behavior of immediately calling weakref callbacks when an object is dereferenced. This is why keeping test suites running on pypy can be difficult, because it has the latter behavior.
> Removing the GIL will require a new ABI, so existing C-API extensions will minimally need to be rebuilt and may also require other changes. Updating C-API extensions is where the majority of work will be if the PEP is accepted.
as you note, there will be *two* versions of the Python interpreter.
that means every C extension has to be built *twice*. against both versions of the interpreter. Go look at how many files one must have available when publishing binary wheels: https://pypi.org/project/SQLAlchemy/#files the number of files for py3.13 now doubles. not clear if we actually have to have both Python builds present, so I would have like /opt/python3.13.0.gil and /opt/python3.13.0.nogil ? if the gil removal changes almost nothing, why have two versions of Python?
This vastly understates the work involved. Most C extensions and embeddings will require major structural changes or even rewrites. These things are everywhere and are a major reason for python's popularity. A typical financial instition, for example, will have a whole bunch of them, with a morass of python code built on top. Many of these companies took ages to transition to python 3 (and many still have pockets that haven't!). For them to remove GIL reliance from their C extensions is a much bigger ask than migrating to 3 was, and their response to being asked to do it is likely to be blunt.
The fact that only a small minority of developers involved in python directly use the C API (and fewer understand the consequences of GIL removal for them) means the issue tends to be overlooked in discussions like this, but it's the reason a GILectomy will be worse than 2 to 3. In practice it's likely to end up being either a fork or a mode you have to switch on/off, both of which would be miserable for everyone.
The GIL is part of python's success story. It made it easy to write extensions, and the extension ecosystem made the language popular. Every language doesn't have to converge to the same endpoint. Different tools are suited to different jobs. Let it be.
The GIL was one of the reasons why the C module ecosytem took off.
It could be a real headache for any hoped-for transition to nogil Python if lots of GIL-reliant C code is floating around where there's little hope of updating it without having to worry about subtle bugs popping up. And even if the conversion was risk-free (which I doubt), many organizations will still not want to dig into their legacy C codebases and make significant changes.
> In practice it's likely to end up being either a fork or a mode you have to switch on/off, both of which would be miserable for everyone.
Yep. That's the plan.
The misery, you mean?
We python users are more than ready for whatever pain we have to deal with in the future, we are used to misery we live it every day.
First, if the extension isn’t being used in a multithreaded environment, nothing should change. Yes, it isn’t thread safe, but it doesn’t really matter in that context. And given how bad GIL-Python works with threads, I doubt the majority of extensions are written for multithreaded applications.
And the ones that are written for multithreaded are probably already releasing the GIL for long running computations, so they should be written with at least a little bit of thread safety in mind. And in the worst case it shouldn’t be too difficult to hack the code to include a GIL-like lock that needs to be held by library users in order to ensure thread safety without really changing the architecture of the extension.
If an extension doesn't advertise that it is thread safe, the GIL get activated at runtime so it will work exactly like it does now.
For example, if you need to handle the external programs input, stdout and errout, most of the examples you find by googling use threads for this.
A lot of those important "extensions" are actually C and C++ libraries designed for parallelism in their native use which have been made available in Python via bindings (e.g. Pytorch). The cores of these libraries are fine, only the binding layers (often automatically/semi-automatically generated) may need to be updated. I suspect only the C-API extensions designed from scratch to only be extensions are going to have major problems with a no-GIL world.
> Every language doesn't have to converge to the same endpoint. Different tools are suited to different jobs.
People have decided they want to use Python frontends. Plenty of alternative languages could have won out, which would have provided native threading, faster runtimes etc, but they didn't; we are stuck with Python. The continued existence of the GIL is incredibly restrictive on how you can leverage parallelism both in Python itself and in extensions. Only in the simple cases can you make the Python<->extension boundary clean: the second you want to be able to call Python from extension code (e.g. via callbacks or inheritance) the GIL stands in your way.
No, they choose it for the ease of using its performant extensions. And those extensions are fundamentally limited in performance by the existence of the GIL. And the authors of those extensions (and their employers) are behind the work to get rid of it.
There are three groups here:
1. "Pure" Python users, whose code makes little/no use of extensions. GIL removal is an immediate win for these uses.
2. "Good" extension users/authors, whose code will support and benefit from no-GIL. GIL removal is an immediate win for these uses.
3. "Bad" extension users/authors, whose code doesn't/can't support no-GIL. GIL removal probably doesn't break existing code, but makes new uses potentially unsafe.
Maintaining the technical debt of the GIL indefinitely so that group 3 never needs to address its own technical debt is not a good tradeoff for groups 1 and 2 which actively want to move the language forwards.
The above arguments could apply to any breaking changes that a host wants to implement. Those who don’t directly consume the change, those who can adapt, and everyone who is fine with the status quo.
Python is free to do as it pleases, but this breaking change is going to result in a lot of churn.
> GIL removal is an immediate win for these uses.
As a minor aside, the GIL free Python version was actually a performance regression, somewhere between 5-10% impact. https://peps.python.org/pep-0703/#performance . So, an immediate loss for nearly everyone.
But besides being incorrect, the idea that extensions that break on removal of the GIL are "bad" is irrelevant. The question is what the ecosystem will bear. The people who own the codebases I'm talking about are not pushing for GIL removal. They have large, working codebases they're using in production and the cost of redesigning large chunks of them would be real. GILectomy is being pushed by people with varied motives but they certainly don't speak for everyone and their case isn't going to be made successfully by casting aspersions on those with different priorities.
What will happen is that extensions that don't work with the new behavior will be assimilated by the proponents. Support for the new behavior will be hacked in, probably with many mistakes.
The bad results will then be marketed as progress.
keeping a gil when entering and exiting c extensions seems like an obvious steppingstone.
How many people do you think would complain if the python devs. replaced numpy and pythorch with pure pyhton implementations for an April fools joke?
Python the runtime may not be the most performant by itself, but python the ecosystem only took of because of performance hacks like those popular modules, pypi, cython, ... .
> Those modules are largely written in C, not Python.
Yet people choose C to implement those modules because Python is not performant enough and isn't that a nice thought? Lets do something good for humanity and drive another stake into that old abominations heart by making it less relevant.
The issue with python is that the ecosystem is so big, everyone thinks their niche is the one python was "built for". Data science, web backends, sysadmin, app scripting, superclusters, embedding... They all think they're the biggest dog, when the reality is that they're more or less all the same. Pypy and cython are probably popular in some of those niches, but python is not just about them.
The reality is that python "took off" because of two things:
- the syntax
- the ease of interfacing with other worlds, typically (but not limited to) C/C++
Removing the GIL seriously threatens the second reason, because integrations that used to be fairly easy will have to be fundamentally rearchitected. Moving from Python 2 to 3 was trivial in comparison, and it took a decade; this transition might take much longer, or never really happen, with massive losses for the ecosystem. And all for what, some purist crusade that interests only a fraction of the whole community...?
Why?
Once you have a threaded Python, you can allocate one thread to run with a GIL and the other threads without it. Old extensions can access one thread with the old GIL API and new extensions can access all the things with the GIL-less API.
People who want the performance will rewrite their extension. People who don't, won't.
Sadly, you can't just trivially bolt on thread safety to code that was not designed with it in mind.
However, if you go the other way and introduce e.g. an @nogil decorator, similar to that seen in some cpython alternatives, people will have a straightforward path to opt in incrementally as they fix and verify the critical parts of their code, while preserving known working behaviour elsewhere without having to throw entire complex systems over the wall at once.
But, if you want free-threading but you have to use an old extension that is not GIL-less, then you still can, by garding all access to this extension with a lock and setting the PYTHONGIL variable to 0.
But, you will need to compile the extension with --disable-gil or to obtain that from somewhere, if the extension author doesn't provide it, as the ABI of "standard" python and "--disable-gil" python is different.
How --disable-gil builds of python and of extensions will be distributed is not yet totally clear. I don't think pypi has already a clear idea of how to handle those extension, for instance.
Basically there will be two build mode.
The standard build that you get at python.org will work exactly as now (so with a Gil).
The disable-gil build (which will most likely be available through conda or similar) will have the option to run either with the GIL or without. It will automatically detect extension that are not compatible and switch to Gil mode but you can override this with an environment variable.
Could you elaborate on how the Python ecosystem will suffer if these financial institutions get left behind? What sorts of contributions come from these users in particular?
StackOverflow answers, side projects, fanboys
If it doesn't work for you, use the GIL enabled. What's the problem of letting others disable it?
How many people will be surprised / upset that they have to do that once `--disable-gil` is mainlined into the default builds on python.org?
How long before 'build it yourself and use --disable-gil' becomes 'we've enabled --disable-gil by default for Great Victory for Most Users'?
I think it's very very naive to think that the flag will remain an obscure 'use if you want to build python yourself' if the PEP is successful.
> The global interpreter lock will remain the default for CPython builds and python.org downloads.
Is a joke. It means, for now. For the work in this PEP.
I mean, you've literally got the folk from numpy saying:
> Coordinating on APIs and design decisions to control parallelism is still a major amount of work, and one of the harder challenges across the PyData ecosystem. It would have looked a lot different (better, easier) without a GIL.
That isn't 'for a few people who are vaguely interested in obscure technical stuff'; this is the entire python data ecosystem, saying "this will be the default in the future because the GIL is a massive pain in the ass".
> If it doesn't work for you, use the GIL enabled.
Sure, it can be opt-out; but you're fooling yourself if you think this is going to be 'opt-in'.
It will be the default once it's completed, sooner or later, imo.
...and when the 'default' build of python breaks c-extensions, that is a breaking change, even if technically you can build your own copy of python with different compile flags.
brew install python@3.12-mp
apt install python3.12-mp
Two major parallel incompatible implementations of Python, maintained at the same time, duplicating all efforts at testing, dev, build, release train, etc. while both are maintained.
Does it sound familiar?
Even ignoring the obvious parallels to Python 3000, there’s a limited amount of time people will be bothered maintaining both.
Which doesn't mean there will be no GIL. Contrary to popular belief the goal of PEP 703 is not to remove the GIL, but only to be able to disable at runtime if you need free threading, so it is an opt-in feature.
In the interim, it seems that Conda have volonteered to build the extensions that are frequently used in the scientific community.
Sam claims otherwise: "Most C API extensions don’t require any changes, and for those that do require changes, the changes are small. For example, for “nogil” Python I’m providing binary wheels for ~35 extensions that are slow or difficult to build from source and only about seven projects required code changes (PyTorch, pybind11, Cython, numpy, scikit-learn, viztracer, pyo3). For four of those projects, the code changes have already been contributed and adopted upstream. For comparison, many of those same projects also frequently require changes to support minor releases of CPython."
https://discuss.python.org/t/pep-703-making-the-global-inter...
In the following talk, he shows the scikit-learn changes that were needed:
https://youtu.be/9OOJcTp8dqE?t=1958
I'd also like to point out that the GIL does not inherently make C extensions thread safe. It only protects the interpreter itself. For example, calling in to the C API can already release the GIL as I explained in reply to /u/bjourne here:
Python is a very slow language whose main advantage is relatively cheap and easy FFI with bindings to all imaginable library already written.
With breaking FFI you'll lose all the libraries while adding SMP to still slow language. It's a cleare loss of value.
In this particular case, I understand that removing the Gil would create potential new risks ( but provide better performance) and although tooling is mentioned, it’s not officially part of the plan, so it feels like a regression of some sorts.
That is unless in the serial execution you still have to pay for some resources that are unutilized.
I guess you could say the world went through a dynamically typed loosy goosy phase and then realised that wasn't so great.
I think one thing that actually has changed is that SMT solvers have massively advanced in the last 20 years to the point that formal verification is practical. Ish anyway. It still seems to require a PhD to do software formal verification. Hardware formal verification is easy though.
The primary disadvantage of the multiprocessing module is it uses separate processes which means my resource heavy Python extension needs to be duplicated multiple times in memory (at least on Windows). I think the move to supporting threads means the module could leverage them instead of separate processes, avoiding the need to duplicate C extensions in memory, and thereby reducing the overall memory pressure on a system.
There are ways around this, Python has a rather rudimentary and archaic shared memory subsystem[0] that can make this much more efficient.
On the other hand, the Python multiproccessing sync primitives actually work over a network with almost no code changes, so you can horizontally scale with little effort... but then you're paying the serialization and network overhead, so ymmv.
[0]: https://docs.python.org/3/library/multiprocessing.shared_mem...
This is documented: https://docs.python.org/3/library/gc.html#gc.freeze
For details, see "Copy-on-write friendly Python garbage collection" by Instagram Engineering: https://instagram-engineering.com/copy-on-write-friendly-pyt...
Edit: Ah I see, you're saying the library itself discards anything created from python. That sounds like an implementation problem then, I'd expect it to at least keep imports so any static code pages don't need to be recreated (the interpreter itself should be fine at least, unless it's doing something really dumb).
Meanwhile, the Steering Council is dragging their feet, even though so many people in the community have chimed in their support for nogil being a thing (in the linked thread and two others).
https://github.com/python/steering-council/issues/188#issuec...
> The PEP was posted five months ago, and it has been 20 months since an end-to-end working implementation (that works with a large number of extensions) was discussed on python-dev.
I would expect the SC to be active participants in the community, including the python-dev mailing list, so they should have known what this change means by that point.
The issue is all the libraries and packages that were built around it. It's now been YEARS of building on top of the GIL.
So anyways, PEP 703 is a a nice effort, but I doubt we (everyday mortals) can enjoy it. Meta, and big companies with a specialized team might be able to exploit it by making sure their entire stack is GIL-free.
EDIT: My bad, the talk is from Larry Hastings: https://www.youtube.com/watch?v=P3AyI_u66Bw
I'd suggest we can be more optimistic. There is a lot of python, regular users write daily, where we delegate orchestration to established libraries - asyncio/web-frameworks/pytorch. The GIL limits how much they can parallelize your code, and its removal will help with that.
Not saying he isn't wrong, though.
Not really. Most of the popular libraries will have nogil versions ready for sure (numpy, pandas, pytorch, etc). Server applications will probably benefit from being able to serve multiple requests in a real multithreaded environment without many changes (in fact, I bet a lot of HTTP libraries will see simplifications due to not needing to deal with processes for real parallelism). Even packages without changes will probably have at least a small degree of thread safety (for example, two instances of the same object should be independent and could be used in different threads) that allows users to still leverage parallelism.
The subinterpreters route sounds really promising since it strikes a good balance between allowing those that need same-process parallelization to be able to do so and keeping the status quo (like workers in Javascript).
I'd actually veto PEP 703 at this point, since the Python optimization team is making good progress, and this could actually take them back a bit, while diverting too much energy into a low-reward avenue.
“Python 4, at this point whenever it’s mentioned in the core development team, it is very much as a joke… We’ve learned our lesson from Python 3 vs 2, and so it’s almost taboo to talk about a Python 4 in a serious sense.”
[...]
“I normally talk about that as a mistake, because Python was more successful than the core developers realised and so we should have been much more aware and supportive of transitioning from Python 2 to Python 3”
[...]
“I’m not thrilled about the idea of Python 4 and nobody in the core dev team really is – so probably there never will be a 4.0 and we’ll just keep numbering until 3.33, at least”
[...]
“We now have a strict annual release schedule, so after 3.10 will be 3.11 and after that will be 3.12, and so forth. We can go up to 3.99 before we have to add another digit. Adding another digit is not completely trivial, but still much better than going from 3 to 4."
[1]: https://peps.python.org/pep-0703/If as a community we invest in those tools and make them easier to build, the cost of upgrading goes down and the velocity of high-impact changes can increase.
JavaScript evolves quickly but so does Python!
(Note that your approach is exactly what they said in the 2 to 3 transiton btw with a special tool that didn't work too well)
But I think they're referencing the litany of transpilers and repackagers which exist for js. So you can add new features and then still have it run on really old systems like internet explorer 9 if you need to.
This has problems obviously and in my opinion for python it would be preferable.
My reasoning being that if you need your code to work on an older system being able to write and use current syntax is preferable to not, and the hard bifurcation that python did with 2 to 3 and now potentially with 3 to nogil seems to me just to break apart the ecosystem more.
https://github.com/facebookarchive/codemod
It sounds like what you are describing is what’s known as poly fills which convert code into a variant that maximizes function across implementations which isn’t really applicable here.
Ironically this is also from Meta which would be contributing to this space increasing the expertise of achieving this result.
For example, is this code thread-safe?
foo(int* x) {
int z;
for (int* y = x + 1; y != 0 && *y < *(y - 1); ++y) {
z = *y;
}
return z;
}
Do you know the reason?
The FAANG etc failed me for anything open source sponsorship related.
Their business depends on it, they made a fortune out of it, yet they donate little or nothing to it, or worse, they forked it and was reluctant to upstream anything.
pick Amazon randomly, it went from exploiting to rip-off in my opinion, thanks to its boss , who sails on a luxury yacht and asks all his employees to be frugal. they all look similar to me as far as supporting OSS goes.
Next.js exists too which mashed up PHP/Ruby style with React.
It was their authors choice first and foremost to publish them as OSS
There is no legal obligation per-se. Open Source is a charity, but you can understand a lot about a company's motives if you compare their FOSS usage to their FOSS contributions.
Goodwill might work for individual contributors, but it’s not enough of an incentive for organizations.
It _should_ be. There's nothing stopping organizations from doing good things except expectation.
There are real costs and that has real consequences. Expectation is way down on the list.
It’s clearly not perfect calculus since so many defect from OSS, but there’s also clearly some business sense there.
Then it just transitions to a charity that everyone relies on. It's been embarrassingly difficult for people to admit this vis-a-vis Linux and the like, but it's obvious in a situation like programming languages. We rely on the enduring charity of it's authors, license-holders and distributors to write and distribute software.
"Giving back" is not an inherently cynical process. Meta has no obligation to publish these patches in the first place, much less work as a part of the community to solve a common issue. But they do, and it's just kinda part of their culture at this point. Yes, they do still gain from it; but so do their direct competitors. It's difficult to characterize as a purely business-motivated decision, because there are much greedier avenues to pursue if that's their MO.
It is all about knowing individual engineers inside these companies. Nearly any engineer in FAANG can, with a little effort, donate their own work time to an opensource project in the form of submitting patches.
The project can help in various ways. Make sure licensing stuff is in place so that whatever IP review process the company has goes smoothly. Make sure PR's are reviewed fast. Delegate responsibility ('hi, would you like to be maintainer of this feature?'). Make sure contributions are publicly recognised (ie. Public page of maintainers/biggest contributors).
Remember that most engineers inside big companies are trying to generate good content for their performance review. "Wrote and open source some code" is usually one of the checkboxes.
I know at least one of those letters that does not want you doing that without jumping through hoops and getting prior approvals. I will let you guess which one that is.
Meta has its own production fork of CPython, Cinder:
There's a whole Silicon Valley episode on this.
Or do license terms apply differently if you’re Meta?
Assuming the company doesn't have clause to not allow contributing to open source. These companies should be publicly shamed if they use any open source code.
I had no problems with this by the way, just saying maybe he can contribute 0.001% of his profit to those OSS projects that he depends on, it will help him to build more yachts too.
One can argue to what extent Bezos is still ultimately "the boss" at Amazon given that he's not the CEO any more, but...
Chromium underpins nearly every major browser that isn't Safari or Firefox. And even more narrowly, V8 powers Node (and Deno).
More Google contributions: Angular, Kubernetes, gRPC, Golang.
Microsoft: C#, TypeScript, Language Server Protocol, VSCode.
Meta: React, zstd, Apache Cassandra, LLAMA.
This isn't even getting into open source patches (Google has consistently been one of the top 10 Linux kernel contributors for the last decade), systems papers that have inspired research and other systems (BigTable, Dynamo, MapReduce, Colossus, Spanner), standards work (HTTP/2 and HTTP/3 were both adapted from Google technology), security vulnerability work (Project Zero).
(The above list is definitely Google-skewed because that's what I know.)
Chrome, though, is a kind of mixed bag. It already has too many features, most likely the reason it underpins so many other browsers is that nobody can keep up with the rate at which Google shovels shit into there.
They should slow down. It would be nice if they could just spend a couple years working purely on stability/performance/security, but alternatively they could just stop, it would be better for the ecosystem.
What FAANG does not do is support critical yet underappreciated Unix infrastructure. By this I mean directly donating to developers without trying to take over the projects.
The quality of FAANG code, especially in the fault-tolerant machine learning space, is questionable as well. I'm not sure why so many commenters (in other comments!) attribute near magical qualities to any output of FAANG.
FAANG are not charities. If they are sold a license for the software for $0, then they don't have much of reason to pay for it.
People apparently forget the amount of money that was and is injected into them.
Someone has to put the money on the table to attend ISO meetings, buy the standards, implement the features into existing C and C++ compilers, for example.
Many of Google's OSS projects are there to help its profits, if not, Google could stop or kill it at will which is fine and is 1 millions times better than closed code, still not the same as projects like python etc.
Do you think they are irrational?
there is graphql, rocksdb and folly. they also open sourced their js engine for react native too.
maybe a good criticism is we depend to much on these companies free stuff lol
I know it is not a popular opinion but the AGPL is there for a reason. If you release your code under a permissive license you should not be surprised that others benefit from it - sometimes massively. That is how it is supposed to be. You can still lament that it is unjust and I feel you, because in a sense it is. If it really bothers you, the solution is easy. Next time choose the AGPL.
Meta and Google have both contributed an enormous amount to open source (creating PyTorch, TensorFlow, Chromium, contributing to clang/LLVM, Linux kernel, HTTP standards), but you're mad because they didn't sponsor your work and a bunch of people mindlessly upvoted it.
> If there’s one lesson we’ve learned from the Python 2 to 3 transition, it’s that it would have been very beneficial if Python 2 and 3 code could coexist in the same Python interpreter. We blew it that time, and it set us back by about a decade.
So it's not so easy to decide to go with again 2 different versions of Python.
I understand how caution they are.
However, this commitment from Meta, complemented with MS financing Guido, and the quality of Sam Gross proposal makes me optimistic.
One core dev for 3 years. Hardly earth-shattering, if compared to the billions poured into JavaScript.
This means multiple devs over a period of less than 3 years.
Jesus. Imagine a language runtime being this hamstrung.
When it came out, this was in store:
https://img.redbull.com/images/w_3000/q_auto,f_auto/redbullc...
What was not in store were:
- multi-core CPU
- wide spread internet access
So people solve concurrency with a simple but efficient and sufficient method at the time: the GIL.
And nobody planned for the scientific and ML stack.
Their team wants it, and is offering hours to help get it done. Is this not what everyone wants from businesses using OSS?
Most of the time in an interpreter you’re iterating through the byte code for a function, and mutating variables. It doesn’t matter if two threads are executing that same byte code, it only matters if they are mutating the same object.
That’s not to say there aren’t some operations that need coordination across all threads. For that you implement safe points so you signal that something has to be done, wait for all threads to reach a safe point, do the work, and then release the threads again. You can often be even smarter about this and avoid stopping all threads at once but instead stagger things.
Allocation and GC you can make more thread local to avoid stopping the world too often.
Without the GIL, you'd still need to use different locking/synchronization mechanisms to ensure thread safety (if required), but they'd be more nuanced and spread over the entire codebase (complicating the backend).
I know Clojure is often described as a dynamic language and that it has some concurrency primitives, but my knowledge ends here.
[1] Just like Python, Ruby does have implementations that allow multi-threading.
This is a great step forward.
I honestly don't think the community is ready for this change and don't expect to ever see stock cpython drop the GIL--perhaps there will be a flag to selectively operate a python process without the GIL (and leave the onous on you to completely test and validate your code works with potentially new and spooky behaviour).
From a python perspective it wouldn't necessarily be a big change, but everything can branch out to c, and there you're going to get in trouble with shared memory.
C extensions have always been able to use real threading. But now there is no GIL to synchronize with the python interpreter on ingress/egress from the extension.
https://realpython.com/python-gil/#what-problem-did-the-gil-...
When I say GIL specific behavior, I mean "python code that specifically requires a GLOBAL interpreter lock to function properly"
Not something that simply requires atomic access or any of the garuntees that the GIL has this far provided, but like, specifically code that requires GIL like behavior above any CPython implementation details that could be implemented with more fine grained concurrency assurances?
I've seen some really cursed python in my days, like checking against `locals()` to see if a variable was defined ala JavaScript's 'foo in window' syntax (but I suppose more portable), but I can't recall anything specifically caring about a global interpreter lock (instead of what the GIL has semantically provided, which is much different)
They are relying on behavior of a single environment. We similarly see a lot of issues moving threaded code to a new operating system in C/C++, because the underlying implementation details (cooperative threading, m:n hybrid threading, cpu-bound scheduling, I/O and signaling behavior) will hide bugs which came from mistakes and assumptions.
Literally speaking, that's exactly what "removal" means. As far as I can tell, GP was wondering why there's so much discussion about replacement, since simply removing the GIL wouldn't break single-threaded code.
Personally I'd much rather see this effort go towards a new language which "feels" like Python but adopts more of the development experience of Go and Rust. From my tinkering it seems like Nim might already be that language, in which case what is needed is investment in its package ecosystem.
The problem with Python 2 to python 3 is that python 3 was essentially a new python 2 esque language instead of just being a major version bump. If python 4 was no GIL python, then both Python and C would remain unchanged as a language.
So, library designers will be under pressure to update their libraries to enable support. We could also have code tools that detect patterns that aren't GIL safe and throw out loud warnings
Python is a single-threaded language. That’s part of its DNA. The community has already been through one traumatic transition in recent history and the appetite for another one is low.
Library authors should not update their libraries to support multithreading, rather the people who want that should be forced to rewrite their code in a language that is more suitable for the problem they want to solve.
Would removing the GIL be a big change for CPython? Yes. But IMO it's worth it
Boooo. Maybe if all you do is ops scripts, but for those of us in data science this couldn't be further from the truth imo.
I'm not saying the syntax is easy for asyncio, or anywhere near as nice as golang or even kotlin is for concurrency, but it's definitely workable in a concurrent environment.
If the changes proposed in this PEP go through, that will no longer be the case. So library authors pretty much will have to either update, or see their modules wither the same way they would if they weren't updated as new Python 3.x versions come along.
> rather the people who want that should be forced to rewrite their code in a language that is more suitable for the problem they want to solve
The vast majority of the Python developer community WANT Python to support true multithreading and being able to solve these problems. We expend inordinate amounts of time and skill trying to make Python work around the GIL, eg. by utilizing multiprocessing.
We want Python to stay relevant long-term. In an age of abundant multi-core platforms, and workloads that can utilize them, the GIL is a major obstacle to that desire.
Err what?
import threading
from concurrent.futures import ThreadPoolExecutorThis kind of garbage logic lead to the horrible Ruby/Python/Javascript with C extensions split that makes their ecosystems very brittle, versus Java/C# where it is expected that things are fast enough without C and package management is much easier.
I understand the desire/demand for general purpose tools. The thing is, there are always tradeoffs. Acknowledging the tradeoffs and designing more specialized tools that work well together isn't necessarily garbage logic.
I mean, it's specialized for:
* data science
* AI/ML
* webdev
* DevOps tools
* an embedded extension language for 100000 programs
...
The sooner core Python devs accept this, the sooner the right thing can be done.
Python is not AWK.
Its literally not, though, even with the GIL.
I imagine there would need to be some kind of annotation to enable the GIL for a method and all of the code it calls, including libraries, so performant Python can take advantage of the lack of a GIL but old code doesn't break. Then all you need to do to maintain compatibility is to annotate your main() and your code should remain compatible for a while.
After all, the referenced PEP explicitly calls for making the GIL optional, not for removing it completely.
That wasn't the problem. The problem was not giving people the ability to make their code python 3 compatible while they were still stuck with python 2. The python 3 interpreter should have had a python 2 mode that gives you warnings.
There were compatibility wrappers for porting Python 2 code to Python 3 but in some instances that was just never going to happen.
This does make me wonder, though, if gevent will survive PEP 703's massive changes to internal Python assumptions. That said, gevent does work on pypy, so there's some history with it being flexible enough to port. Hopefully it won't be left behind, otherwise codebases using gevent will see this as a Python 2 -> 3 level apocalypse!
But this only solves network concurrency problems.
Very reasonable tradeoff for an application server, which is embarrassingly parallel and holds little state.
The GIL makes python code have cooperative threading. It does not protect from e.g. your thread's view of state mutating when you make a database call.
I also believe it is best practice not to mutate data without holding the GIL in extension code, not a requirement - but I have mucked with a lot of different extension API so I might be confused.
While C extensions can release the GIL, that only makes sense if they do enough work that a Python thread could get some things done in the meanwhile, and it wouldn’t be surprising to the caller. Obviously the C thread can’t interact with the Python world after the GIL has been released.
Some things really are too big to change.
Then transpile Py3 code into Py4.
And it won't change anything about c extensions.
I'll admit that, yes, changing string to bytes and unicode to string was a bit annoying, but the change itself wasn't fundamentally 'of dubious benefit', it did have benefits, and related to this, the only major issue was that you couldn't, for a long time, have code that worked in both where it came to literals. The biggest problem here was the implicit conversion from 2, that I agree needed to go.
Most of the other things can be trivially fixed automatically, or at least detected automatically, but without type hinting, it wasn't really easy to fix the automatic conversion.
There were other changes that were a bit tricky, but the majority of issues stemmed from the str/bytes change.
Worked for ruby. The original interpreter MRI has a GIL too. Rubinius and JRuby added multi-threading with limited amounts of pain and people fixed libraries over the years. Sometimes just sprinkling lock blocks around a particular FFI calls or only doing them from a dedicated thread will do the job.
LOAD_FAST foo
LOAD_CONST 1
BINARY_OP +
STORE_FAST foo
There are a lot of things to be worried about with the GIL conversion of new race conditions that could happen, but there's already too much misinformation out there about the GIL, let's not spread this one even further.
My guess is the GIL or smaller mutexes will be needed for C extensions and a few other areas, but it’s also likely that could be moved to an opt in mechanism over time.
Not really, just make those operations atomic or have automatic locking
Except for every single piece of python I have ever written which will continue to work just fine.
I fail to see why this is a big deal. Don't big tech companies contribute much more to python FOSS
i would gladly accept a python3 to python4 breaking change where all the necessary bits are changed to get real multi-threading (no-gil, gilectomy or whatever).
it's just so worth it.
insert bell curve meme
The entire debate seems to be for code that strikes me as being far and away from the typical Python programmer.
All the ML code is written in lower level languages and that’s very unlikely to change, GIL or no.
Instagram (and presumably threads) runs on Cinder which is Meta's fork cpython with GIL removed.
> support in the form of three engineer-years [...] between the acceptance of PEP 703 and the end of 2025
It seems much more like them pledging engineer time from Meta employees to work on this project than sponsorship, not a monetary sponsorship like the title implies.
"it would be great if Meta or another tech company could spare some engineers with established CPython internals experience to help the core dev team with this work."
Regardless, the previous commenter's point stands that the title could be a lot more informative.
Direct link to the comment, where Meta offers to commit to working on this issue for 3 person-years: https://discuss.python.org/t/a-fast-free-threading-python/27...
They use Python. They want to donate engineering time. They think this is a good idea. Why does there have to be a sinister motive?
I don't see any charity here, this is just a smart business move by them. When you run as many servers as Meta does being able to run 1/4 as many instances is huge RAM and therefore money saving. It won't take long to add up to 3 engineer/years.
They have the expertise to run a language fork and because of the scale they run at they would easily save more money doing so. Meta contributing to OSS is not an economic decision. It is about branding, trying to attract technically strong individuals (who tend to contribute to OSS or want to), and sticking to their open culture.
Actually, experience has shown the opposite to be true. Time and time again even large organisations find their fork falling behind the community version and the pain of merging new versions from upstream grows and grows
We all remember the anti-OSS 'Linux is only free if your time is free'.
The alternative is Meta puts in no money to salaries and benefits for up to three individual's time on this project, or Meta puts in no money at all.
Meta is paying, one way or another, for this project.
Did I deny this?
To put it in extreme clear example, just said time for someone to punch you != time for someone to work for you, while money is same all across. I hope you don't disagree with it.
You could maybe make the argument that they're trying to swing priorities if this is considered valuable but less valuable than other things, but given the PSF doesn't actually employ anyone who it can direct to work on specific projects anyway, how is this any different from the already existing prioritisation process of "things that somebody is interested enough to go through the PEP process and then work on"?
Philosophically, Python is close to C in the sense that low-abstraction interfaces to system functionality (file descriptors, memory) exist. Numpy allows access to uninitalized memory and you can manipulate how numpy uses memory (mmap, shared memory, sharing memory between objects). Ctypes allows access to malloc and dereferencing dangling pointers. This is not exactly normal Python, but it's inside the realm of the possible in Python.
All of this makes me worried that the challenge of "free threading" is quite large. I wonder if the subinterpreters solution offers a safer route. At least it's not pulling off the GIL band-aid all in one go.
Also a comment on Github is not a binding support contract. Meta executives could deprioritize this project at any time... hell the person making the commitment could already be laid off, we have no clue. As someone who worked in big tech for a long time, trust me - it needs to be in writing with an exec signature.
This comment is the definition of FUD.
I'd wager the vast majority of programs being written that need concurrency or parallelism still use multi threading or SIMD.
Both things aren't exchangeable, of course!
Thread have direct access to process memory. Other processes usually don't. Big difference!
Most of this work has been on Javascript, which is single-threaded and less mutable than Python, but PyPy has also made some significant contributions. PyPy has a GIL.
There is also work on Ruby. Ruby is even more mutable than Python, but also has a GIL. Work on Ruby mirrors that on Javascript.
Also Carbon is still very much an experiment and not used for much. Google has a lot more Rust in production now, especially in Android and Chrome.
Wouldn’t that still be Python, just a different implementation?
I think the problem there is even a company like Meta wants to rest on the shoulders of the giant Python community. It makes sense for Meta to try to get parts of Cinder into Python.
i really don’t understand the trend with launching a new language every time it seems convenient; it may have the advantage of being made at {company}, but this is also its great downside: it exists, is maintained, and used only with relation to the project/company that prompted its creation
for example, swift is only used for swiftui and some other apple specific stuff, and no sane person is going to write an unrelated serious project in it
Perl never had anywhere near the wealth of libraries and community support Python enjoys today. The language is used nearly everywhere, from utility-scripts to build piplines, desktop applications, webservice backends, all the way up to scientific computing and AI.
Python shines because of it's ecosystem and community support, which I'd guess (without having any numbers to back this up) may very well be unmatched among programming languages today. Maybe Java and C/C++ come somewhat close, but I'd doubt even these behemoths would stand a chance in a 1:1 comparison for sheer size and versatility of what for Python is just a quick `pip install` away.
Any language that want's to replace Python, even internally at a company, will either have to deal with the fallout of leaving all that behind, OR be compatible with Python to an extend that it doesn't matter. And that's a really tall order.