Hacker News Reader

Comment by Animats | Hacker News Reader

This is just a way to do the same thing as "multiprocessing", but with less memory usage. You still have multiple Python instances that send messages back and forth.

I wonder if they ever fixed the CPickle bug which broke it if you were using CPickle from multiple threads.

View on HN

Yeah, it's got some of the same weaknesses as multiprocessing (and several new ones). Conceivably you could provide an API for handing off objects to the other interpreter without copying. I'm imagining an API like:

  my_foo = interpreterX.pass_object(my_foo)

(The assignment being required to delete the originating reference from the source interpreter.) The interface would be obligated to check that there are no references that escape to the current interpreter and then my_foo and all referenced objects could be handed off to the other interpreter in whole.

I don't have any intuitions for if that would be cheaper than copying or not, and getting it right is certainly more difficult than serialization. (Because of the complexity, it's not worth having if it isn't cheaper.)

What in your view are the weaknesses of subinterpreters compared to multiprocessing?

If they live in the same process (address space), then it's a lot harder to correctly manage concurrent memory access (at both low level and high).

It's not a theoretical problem, just a very likely pragmatic observation. (Meaning, we'll see bugs in both the interpreter and the code using this feature.)

Plus if one of the threads crashes, the whole process aborts. (Sure, the interpreter can handle a lot of faults gracefully, but not all.)

The main weakness is the one pas described in the sibling comment. Historically, Python C modules, inside and outside stdlib, have more or less safely been able to assume there was one global interpreter. This means they may have global state (apparently even some stdlib C libraries assume this). With multiple interpreters in the same process address space, any global state in C modules conflicts.

I'm not claiming any originality here — the author of the article recognizes this problem and describes it, starting with:

> Because CPython has been implemented with a single interpreter for so long, many parts of the code base use the “Runtime State” instead of the “Interpreter State”, so if PEP554 were to be merged in it’s current form there would still be many issues.

Less memory usage, and - hopefully - without all the quirks that crop up with multiprocessing. Off the top of my head: subprocesses don't always want to die along with the main process; error conditions can cause the underlying IPC layer to end up in a permanently stalled state.

I think you can avoid these kinds of issues if you use multiprocessing.Pool as a context manager.

It avoids some quirks but also introduces new quirks multiprocessing doesn't have, like broken C modules (including parts of the core interpreter and stdlib) that have global state, rather than per-interpreter state. There's a huge ecosystem of Python libraries in the world and most have been able to more or less ignore the distinction between per-interpreter state and global state prior to this proposal. (Not true if you actually used the C API to embed many interpreters in a process, but most people don't do that.)

Is it possible to load an instance of a native library per interpreter? Give each one its own memory space?

That requires making the native parts of the stdlib position independent (relocatable). I don't know what obstacles are there currently, but basically you'd need to have a runtime linker that gives a reference to the loaded object/module to access the symbols, but these are usually handled by the C compiler and linker. So it's doable, just a lot of effort.

Theoretically, if this proposal lands, all of the Python stdlib will have to be fixed not to have global state — the main concern would be 3rd party C libraries, I think.

I don't think you can do this with traditional dynamic linkers — the separate memory space is not so difficult (requires relocatable libraries, -fPIC, which is usually already enabled on ASLR systems) — but you would want to be able to load the same .so twice without symbol naming conflicts. I don't think most dynamic linkers (ld-linux / rtld-elf) support that. I could be mistaken, I am not very familiar with any implementation.

There is nothing preventing you from adding this support to an existing dynamic linker and using it for your program, though.

> but you would want to be able to load the same .so twice without symbol naming conflicts

This is supported in ld-linux by using the dlmopen() glibc function and distinct namespaces. Loading the same .so file multiple times is one of the use cases explicitly mentioned in the manual page[1].

[1]: http://man7.org/linux/man-pages/man3/dlmopen.3.html#NOTES

Thanks, I wasn't aware of that!

One caveat seems to be:

> The glibc implementation supports a maximum of 16 namespaces.

Reply on news.ycombinator.com