Hacker News Reader

Comment by colanderman | Hacker News Reader

The whole Python versioning fiasco is simply baffling to me as an outsider. Why not mark Python 3 modules with a tag, or different file extension, or anything, and require the Python 3 interpreter to be able to interpret Python 2 code as well as Python 3 code?

This is e.g. how the (roughly analogous) split between C and C++ is handled, and it works fine for the most part. No bizarre polyglot code games.

Edit: Just to be clear, C++ is not a superset of C: http://stackoverflow.com/a/1201840/270610

View on HN

Are you expecting the same running interpreter to have both python 2 and 3 modules at the same time, with calls between the modules working? You can easily keep things separate - python 2 executable is named python while python 3 is python3. https://www.python.org/dev/peps/pep-0397/

Sadly concurrent running in the same interpreter won't work because Python 3 has different types for strings (unicode only) and a bytes type, while Python 2 has a str type (pretty much like bytes but can also be treated as a string) and a separate unicode type. Python 2 had code that tried to do the right thing automagically with str, such as automatic promotion to unicode where it looks like you really meant a string and that is what is needed. Automatic promotion can fail, be unexpected, is hard to test and numerous other issues. That is why Python 3 was needed and why it needed to be incompatible. To have Python 2 and 3 code interoperate in the same interpreter would require another layer of automatic promotion/demotion and heuristics when data goes between the two.

C++ was explicitly defined to be a superset of C so it doesn't have these kind of issues. The analogous situation would be if C didn't distinguish between long and pointer types, and C++ did. You'd need rules for how to convert between the two worlds, trying to combine or disambiguate the types depending on which way types are going. It would be a mess.

But imagine what would happen if they had created a compatibility layer. Yes, it would be very hacky, and require a lot of seemingly gratuitous work - tons of standard library functions would have to check the compatibility version of the caller and change behavior accordingly. And the goal of cleaning up the codebase by removing things like old-style classes wouldn't be achievable. But what if that work had been done, and the mess accepted?

Then Python 3 code could use every unmodified Python 2 library (albeit potentially requiring the former to do some messy conversions between str and bytes when making calls), including the long tail of legacy libraries that will never be upgraded. Meanwhile, once the issues in the first Python 3 release or two were straightened out, there would be very little reason to use the Python 2 interpreter, since the Python 3 one would be a drop-in replacement; so migration to the latter would be relatively swift, and it wouldn't be too much of a burden to ask someone to upgrade as a prerequisite to using your code. So there would be very little reason to write new Python code in the Python 2 dialect (unless you really really hate making print a function :), and existing code could be migrated one file at a time. Including popular libraries that wanted to maintain backwards compatibility with Python 2 programs - they would have to add compatibility checks similar to the standard library, but could otherwise go full Python 3, rather than having to use the awkward intersection of language semantics and compatibility libraries currently required.

With the transition so much smoother overall, native Python 3 code would probably take over relatively quickly. (Maybe it would have even been feasible to remove the Python 2 compatibility layer eventually, though probably not.) Whereas in reality, today, over 7 years later, Python 2 is still more popular than Python 3 [1].

[1] https://lwn.net/Articles/640181/

Yes, that is what I expect. The various flavors of C and C++ get along just fine that way.

C++ is not a superset of C: http://stackoverflow.com/a/1201840/270610

C and C++ have a similar issue with strings. C code uses NUL-terminated char arrays, while C++ generally uses std::string. If you want to interface the two, you need glue code (written in C++).

Surely Python 3 can work the same way: to share strings between a Python 2 and Python 3 module, you need to convert them using glue code written in Python 3. Sure it's messy, but writing polyglot code is messier.

Except that you can easily go from std::string to cstring and back in the same program. And many of the standard library functions work fine on c strings (probably over half of <algorithm>), etc. IMO theres no compatibility issue between std::string and c strings.

I wonder if there's even a reasonable way to do this in Python. In C, strings are data that you're explicitly passing around, but in Python, you are constantly calling __getattribute__ under the hood just to execute your code.

I don't know that it would be impossible to make that all work, but I wouldn't assume that it is.

You don't need to actually convert anything: Python 2's str can be identified with Python 3's bytes, and unicode with str. You 'just' need to have their __getattribute__ methods, along with a ton of standard library functions, check the Python version of the caller and change behavior accordingly. Ugly, but I think it would mostly 'just work' for user code.

Actually C++ is not a superset of C. And it does have these kinds of issues. C++ continued to diverge with C99/C11. https://en.wikipedia.org/wiki/Compatibility_of_C_and_C%2B%2B

Function pointers are one scary area. I don't know if C++11/C++14 decided to fix this yet. Most people relied on undefined behavior to make this work.

Since we are in pedant's corner, note that I very carefully used past tense. Yes it originally was a design goal.

Source: The Design and Evolution of C++, by Bjarne Stroustrup. (I don't know which printing I have.) Page 120, Section 4.5 "No gratuitous incompatibilities with C". It explains my original claim, and also why it couldn't be 100% compatible due to some other C++ desirables (eg better type safety). Partial quote:

"C++ doesn't aim at 100% compatibility with C because that would have compromised the aims of type safety and support for design. However, where these aims are not interfered with incompatibilities are avoided - even at the cost of inelegance. In most cases, C incompatibilities have been accepted only when a C rule left a gaping hole in the type system."

> python 2 executable is named python while python 3 is python3

Nope, not quite. Python 2 is named python2 and python 3 is named python3, on every system I am aware of. 'python' is ambiguous and can be linked to either [1]. I know at least on Arch Linux, python links to python3, and I have run into a handful of scripts that break by assuming python is always linked to python2.

[1] https://www.python.org/dev/peps/pep-0394/

Because you actually can write code that is compatible with both Python 2 and 3.

Also major issues people have with migration is unicode. In respect of unicode, majority of the code is simply broken and fails to work on Python 3 because Python 3 is more strict about it.

In Python 2 you store text and binary data as bytes (no distinction, there is unicode type but it's more hassle to use it and frankly almost no one does use it) while in Python 3 text and binary data are two distinct types.

So if you have code in Python 2 and you did not distinguish what is text and what is binary now you have to go through your entire code and identify all those parts, this is especially hard since Python is dynamically typed language and you don't have type checker to help you with that.

This is also why it is much easier to write Python 3 code and make it run on Python 2 as well than the other way around.

As for having interpreter compatibility layer. First python does not care about extension you use, and unlike some languages it actually allows you to install multiple versions at the same time without conflicts (checked 2.4 all the way to 3.5). You can easily control which version you want to use by shabang, for example:

    #! /usr/bin/env python2

    #! /usr/bin/env python3

    #! /usr/bin/env python3.5

You can also write code that is compatible with both C and Perl. Doesn't make it a good idea.

Shebang doesn't work on a per-module basis, which is what I am advocating. If there were a way to specify versions at that granularity, there would be no need for 2/3 polyglot tricks.

That's a strawman argument. Majority of python libraries are written that way.

I guess I'm incredibly lucky or you're incredibly unlucky, because I did not run into a library that would only run on Python 2. Ok, I did run into some, but those were no longer maintained and frankly I would not use them even on Python 2.

The problem with shebangs used to be (don't know if it still is) that quite a few OS's had python 2.X, but no actual binary called `python2`, just `python`.

Python installs itself as pythonX.Y (for example python2.7) then symlinks such as python2 (pointing to python2.7) and python (pointing to python2) are made.

It's mentioned in PEP0394 that some older distros did not have python2 even though python 2.X was installed. Since RedHat still supports a release with Python 2.4 as the default, I guess it might still crop up on some boxen, at least for another year until RHEL 5 reaches end-of-life.

Assuming python points to python2 will also break stuff, though, there are more bleeding-edge OS's where python points to python3 and has done so for years (e.g. ArchLinux).

I think this is fundamentally because of the Python philosophy around one-true-way to do things (something that's open to interpretation given the number of string functions that are around). IMHO, the Python 3 vs Python 2 split is a phlisophical rift rather than a technical one. There is no real technical reasons why print and print() cannot coexist.

In all reality, Python 2 and Python 3 are completely differently languages at this point. Communities/companies/projects that have adopted Python 2 do not care about anything that Python 3 gives them. Take for instance Flask - to maintain compatibility with the WSGI spec they have to maintain Python 2 compatibility indefinitely. Less than 30% of all software is compatible with Py3 and any new software releases are built on top of Python 2 - yes I'm looking at Tensorflow.

I'm very doubtful that Python Foundation is going to drop support by 2020. Quite simply - a highly funded Python company (like anaconda) can create a fork.

The only way this situation can be fixed is Python 4 with co-existence of Python 3 and Python 2 functions.

As an aside, this:

> the Python philosophy around one-true-way to do things

I've always thought to be a total joke. There's been like 5 different modules each for string interpolation, subprocess management, and argument parsing. Heck, I thought I was on top of which was in vogue and apparently just two weeks ago there's a new "one true way" to do string interpolation.

I came across this recently:

> Barry Warsaw, one of the core Python developers, once said that it frustrated him that "The Zen of Python" (PEP 20) is used as a style guide for Python code, since it was originally written as a poem about Python's internal design. [0]

I did a quick search for a little more background but didn't find anything. I'd be curious to hear more.

[0]: https://github.com/amontalenti/elements-of-python-style#a-li...

I always thought that was mocking with Perl, and nothing else.

Then my analysis that it is a total joke is correct ;)

Looks like so.

Almost all the lines on the Zen of Python are inversion of the Perl guidelines. Some with the exact same phrasing, just with the antonyms of the original.

I think this needs to be the responsibility of things like PEP rather than the actual language. Otherwise you break old code every time you find a better way.

> yes I'm looking at Tensorflow.

Check the very first issue that was reported: https://github.com/tensorflow/tensorflow/issues/1

What do you do when a value passes between major versions which have different implementations of its type? What do you do with the `int` type, which was actually two separate types in Python 2? What do you do when the binary representation of strings changes (as it did in the 3 series) and you want to pass that into Python 2 land? What do you do about extension modules written in C, which many popular Python libraries use to speed up hot code? Do you include copies of both standard libraries, meaning you now have duplicate modules and may get one or the other depending on the tag in your file? Which version of Python 2 do you target? What happens when there's a bug in one Python 2 implementation but not the other? How do you handle having two C APIs at the same time?

You mention C++ as though it worked out great, but C++ is still full of sharp edges and features that interact weirdly with its C legacy, and it has never completely replaced C (especially for libraries). And that's despite having a few advantages that made it possible in the first place, like having a completely different stdlib built on top of the C stdlib, and producing machine code rather than compiling for a VM on the fly.

I've always wondered this. Why not have .py and .py3 extensions, and require the python 3 interpreter to fall back to Python 2? That would also solve the problem of "which version of Python is this script I found written in?"

> require the Python 3 interpreter to be able to interpret Python 2 code as well as Python 3 code

Because that would make the interpreter a whole lot more complicated. A lot of technical debt was cleared with the 2-3 transition, adding it back in defeats the point.

C/C++ is different, C++ is a superset of C. That makes things a lot easier.

Edit: Ok, C++ is not a superset, common misconception.

C++ is not technically a superset of C. Here are a few examples that are valid in C, but not C++:

http://stackoverflow.com/a/1201840/111426

However, it's close enough that I generally agree with your point.

>C/C++ is different, C++ is a superset of C. That makes things a lot easier.

This is entirely incorrect, but I'll give you a break since it is a common misconception. C and C++ are truly two different languages. C is not 100% compatible with C++.

> C++ is a superset of C

No it isn't.

> and require the Python 3 interpreter to be able to interpret Python 2 code as well as Python 3 code?

The interpreter is too messy for that and the transition to Python 3 was not considered well enough. There is no hope on that front. Even if you solve everything else, the C ABI will still be broken.

yeah, the python folks lost their mind around the 2/3 split. source: ex-python user.

Reply on news.ycombinator.com