Comment by jamesdutc | Hacker News Reader

Comment by jamesdutc | Hacker News Reader

Parent Full thread

Embedding Python 2 inside of Python 3 (or vice-versa) is not very hard to do on Linux. Simply `dlmopen` the .so in a new linker namespace and write a little bit of bridging code to interface the two object layouts.

Python has a very rich runtime, so there are some tricky problems to solve if you want this to be perfect. e.g., circular references between the GCs. However, there are simplifying assumptions that can dramatically reduce the difficulty, and these assumptions might not significantly hinder the language-upgrade use-case.

We've known about this approach to embedded interpreters for a couple of years, but we've not found a sufficiently compelling use-case for anyone to develop it beyond a simple proof-of-concept.

In general, it seems like there most compelling use-case for inter-language interfaces is writing core libraries in something fast, (mostly) runtime-free, and portable, especially for codes that no one wants to write twice.

Though I spend most of my time writing codes in slow, rich runtime languages, I've been on the lookout for a better technology to replace C. There are quite a few interesting options, and some have even less of a runtime than (dynamically-linked) C!

> Embedding Python 2 inside of Python 3 (or vice-versa) is not very hard to do on Linux. Simply `dlmopen` the .so in a new linker namespace and write a little bit of bridging code to interface the two object layouts.

Where can I pip install this from?

Or is there a reason that none of the huge numbers of python users delaying their transition as much as possible until all their libraries were updated built it?

> Where can I pip install this from?

A couple of signatures have changed, but the general approach looks like: https://gist.github.com/dutc/eba9b2f7980f400f6287 or https://gist.github.com/dutc/2866d969d5e9209d501a

The above will launch ("embed") a Python 2 or Python 1.5 interpreter from within a Python 3 interpreter. If you use `dlmopen` and `LM_ID_NEWLM`, the guest interpreter will have its own linker namespace. In other words, the guest interpreter (and any DSOs it opens) will be totally isolated from the host interpreter.

The above shows the use of `PyRun_SimpleString`. Since `PyRun_String` accepts a `PyObject* globals` and `PyObject* locals`, you could build a very basic bridge in <30 lines of code by using serialisation to copy and convert. (For user-defined classes, you would need something better.)

I've given a many talks about this at various Python and PyData conferences. (I've mentioned it a number of times on HN, both in response to complaints about the Python 2→3 transition and in response to comments like yours suggesting this solution to that problem.)

Though presented as a joke, the approach could be made to work with some effort. I can't speculate on why no one has ever followed-up on it, and I can't speculate on why the Python 2→3 transition has been so difficult for some users. Perhaps in some places, financial or organisational arguments are more influential than technical arguments.

Reply on news.ycombinator.com