I think the Rationale section tries to explain the benefits? https://www.python.org/dev/peps/pep-0554/#rationale
What I don't get is how they expect C extensions to adapt. I don't expect widespread support for this for like... a decade?
What I don't get is how they expect C extensions to adapt. I don't expect widespread support for this for like... a decade?
The rest of the section is sparse on what you actually do with it. It says that it "has the potential to be a powerful tool" but not what you do with the power. It says it's about "enabling the fundamental capability of multiple isolated interpreters" but it's not clear what the capability brings you. The only thing with some details is the first sentence - "Running code in multiple interpreters provides a useful level of isolation within the same process." But what is the level? It sounds like it gives you isolation for most things but not all, which is about as useful as a face mask with a breathing hole cut out in it.
It'd help to see a concrete answer of something you can build with this that you can't build without it (or perhaps not as easily/performantly/reliably/etc.). The Ceph PR, where they do something similar themselves (https://github.com/ceph/ceph/pull/14971), gives a good answer: "Notably, with this change, it's possible to have more than one mgr module use cherrypy." (But it's still not clear to me why the mgr can't just use multiple Python interpreters... ceph-mgr is kind of an amalgam of various useful services for your Ceph cluster, and it's never been clear to me why it has to be "the" manager with some modules instead of various independent services you can turn off/on as you need.)
I do not think multiprocessing will meet the concurrency needs of many potential applications.
For one thing, I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen. To run Python code in a separate process with some set of initial data, it basically pickles it in the parent and then unpickles it in the child. That seems awfully kludgey to me.
For another, if you have multiple concurrent tasks that have to share data, doing it within a single process between multiple threads is a lot easier than doing it between multiple processes. Every other major language besides Python allows the former; only Python has the GIL which makes the former basically useless.
FWIW, the subinterpreter threading scheme will also use a similar message passing construct to pass values between interpreters. As the sibling comment mentions, it's just a message passing scheme.
That said,
> I've looked at the multiprocessing source code, and to me it looks like a hairball of bugs just waiting to happen.
I have as well, and completely agree.
But in general it's pretty useful.
Everything seemed to be working fine until it didn't.
As noted in a reply to your other comment, this is not true -- Ruby has the equivalent of Python's GIL.
There is currently a project to abstract the C-API with HPy: https://speakerdeck.com/antocuni/hpy-a-future-proof-way-of-e...
The idea is to (very slowly) remove access to direct CPython internals and give a handle to important low-level behavior through a standardized API hidding implementation details.
It's a lot of work, and going to affect a huge part of the ecosystem, but I'm hopeful this is something the extensions authors will actually want to be part of because it promises to solve a lot of problems:
- on the long run, this will make maintaining a c-extension much easier: more stability no matter the evolution of upstream, more certainty.
- there will be a clear way to do things, which means better documentation and conventions, potentially libs to do common things. Right now creating extensions is more a set of receipes people found out to work.
- because of this, scrutiny will be given to common patterns, which will be abstracted and optimized, and given the security of an API contract
- extensions using this API will work out of the box, or faster if they already did, on alternative python implementations like pypy
Of course, the end goal is to free the python core devs to experiment with breaking changes at the c level, such as changing the memory management model, and eventually maybe remove the GIL or allow for a JIT.