How to support both Python 2 and 3 | Hacker News Reader

How to support both Python 2 and 3

ondrejcertik.blogspot.com

84 pointstshepang22 comments

this article takes awhile to get to the point, but the point is correct which is:

> In other words, this is how it should be, that you write your code once, and you can use any supported language version to run it/compile it, or develop in. But for some reason, this obvious solution has been discouraged by Guido and other Python documents, as seen above. I just looked up the latest official Python docs, and that one is not upfront negative about a single code base. But it still does not recommend this approach as the one. So let me fix that: I do recommend a single code base as the solution.

2to3 is definitely useful to do the initial conversion of a codebase from something totally 2.x into one that is 3.x compatible. But from that point on, you need to put it away and not look at it again. Now that it's socially acceptable to target 2.6 at the lowest, libraries that wish to target 2.6-3.x still need to have a little bit of conversion code present, i.e. a subset of six, but other than that the code can be fairly py3k-ish.

Armin's article here is the most up to date discussion of this: http://lucumr.pocoo.org/2013/5/21/porting-to-python-3-redux/ (though lots of us disagree with his decision to dump 3.1/3.2). This article has lots of great advice like "test for py2k not 3k" as well as his metaclass decorator that is a lot nicer than six's. SQLAlchemy's second py3k port (yes, due to the original advice we all have had to port to python 3 twice) makes heavy use of these techniques.

At this point I have all of my major codebases supporting 2.x-3.x. Particularly, Mako templates supports python 2.4 all the way through 3.x with no conversion - and Mako deals heavily with unicode and Python AST structures.

So hopefully the Python docs can catch up with the reality soon. The original advice to think very separately about your 2.x and 3.x codebases was very wrong, has wasted a ton of time (since you end up porting to python 3 twice), and overall slowed down the adoption of py3k, a slowdown that fortunately is finally lifting.

> I'll start with the conclusion: making backwards incompatible version of a language is a terrible idea, and it was bad a mistake.

> So let me fix that: I do recommend a single code base as the solution.

> The original advice to think very separately about your 2.x and 3.x codebases was very wrong, has wasted a ton of time (since you end up porting to python 3 twice), and overall slowed down the adoption of py3k, a slowdown that fortunately is finally lifting.

Is this opinion common? Do Guido and the other core developers admit they were wrong?

> Python 3.0 will break backwards compatibility. Totally. We're not even aiming for a specific common subset.

That a single code base is now possible, albeit "not entirely idiomatic", does support the conclusion that the original aim to break compatibility was misguided at best, and raises questions about the governance of Python overall.

Well zed seems to agree

https://twitter.com/zedshaw/status/364118939233554432

> they've been keeping this "dual mode" option mostly a secret so as to not have to admit they fucked up

If this is not a joke, Zed is paranoid.

If 'raises questions about governance' means the leadership can't predict the future, then I agree. Else, this is not fair, since features were added to Python 2.6 to reduce migration pain (which incidentally enabled this single-codebase thing possible).

I think that the ideas and tips of the article are useful, but I don't like the opening phrase.

People make mistakes. Once a mistake is part of a language specification the only way to solve it is to make a backward incompatible change. It's pain in the ass for the users, so you must not change this kind of changes too often if you like to have users.

A possible alternative is to have a huge metaprograming platform like Racket, that can still run the old version "mzscheme" programs almost unchanged [1]. But in the Racket/Scheme/Lisp word everyone is running their own slightly "improved" version of the language. So multiple languages are more natural than in Python with the " There should be one-- and preferably only one --obvious way to do it." philosophy.

(For example, they changed allowed syntax of "if".)

The commit is more interesting than the article: https://github.com/sympy/sympy/pull/2318

It does fail to mention that you need to also use special exception handling in all cases, eg.

    try:
      ...
    except TypeError:
      e = exception()
      ...

...but it hits all the other things, except the obvious, make sure your tests are running continuously using py2, py3 and pypy, otherwise you'll just frustrate yourself as you're coding.

90% of it is print statement -> function. I got pretty tired of scrolling past all that to try and find the interesting bits.

That exception handling is only necessary if you need to support 2.5 or older. A lot of projects now are just supporting 2.6 and newer, in which case you can use the Python 3 syntax:

    except TypeError as e:

Call me crazy but I would have renamed Python3 to a new name, i.e. Snake. That would have made it crystal clear that it's not the same language.

Seriously, Python3 made me hate python. A couple examples why:

  - The fact that I was using Archlinux and they decided to make "default python" -> "python3" = Hours of wasted efforts figuring which scripts weren't working.

  - Every time I tried to go with 3.x = Screwed because libraries I needed weren't ported

  - print 'x' not working = Just plain annoying

Let's make a new python4 == python2.7 and forget about all this python3 bullshit.

I seriously hope you're trolling. Yes, the transition is harsh, but it's getting better. On your points

1) that's a problem with Archlinux then, not Python. It was dumb to set 3 as default that sound

2) `3to2` works pretty nice for porting code

3) So you have to put parenthesis because it's now a real function. That's so bad. Also you can do stuff like `map(print, myiterable)` which you could not before, what a pain!

"Yes, the transition is harsh, but it's getting better".

Exactly. It's getting better, but it was a hell of an annoying time to be between both. Thus why I've come to hate python. I gave three simple examples of things that annoyed me. You can't just dismiss them like they're not worth anything.

The process is supposed to be simple. I.e. pip install <library>. If you have to clone it, 2to3 it, fix remaining bugs, it's a hell of a pain in the ass.. and just a plain waste of productivity and time. It would have definitely been possible to have python3 backward compatible or slightly incompatible. What happened however is more of a "Let's screw up all existing code to fix a couple things".

To take on your example, yes, using print as a function is cool. But how in hell can't you make it possible to use map(print, ..) in python2.x? It's just about being anal about consistency that print should be a function and not a statement, thus screwing all existing code.

My point is not about if python3 is useful or not.. but more about the way it happened. If you want to make something fully incompatible, fine, but don't do it by screwing everyone who's been using your framework/language/library.

It's like someone deciding that red lights on the road should now be green, and green lights be blue. Let's just change it. Who cares how many accidents will happen?

You might think I'm trolling but I'm not. It's just frustration of literally wasting hours and hours of fixing stuff that was already working (!!!!!!!!!!!!) but isn't anymore. Now, take this time, and multiply it for every production servers that I have running slightly different builds or OS. How in hell something that's been working for 2 years isn't anymore because I updated my os to the latest version? Oh yeah, right, there's a new python version that screwed up every existing scripts. Let's make that the default executable.

You are running production servers on Arch Linux, and then pulling updates which change the default Python to 3 without review?

Sounds like you were asking for it.

Yeah, I won't ever, ever, run production servers on ArchLinux. (Even though I still use arch for my own machine). Doing that is asking for unnecessary pain.

I was used with slackware, gentoo, debian and ubuntu to make frequent update to make sure I had the latest security fixes. On archlinux, it's a different story. "Updating" means "Take a gamble to update the system hoping important packages haven't been changed, renamed or deleted.". Ok, it's not really a gamble because you're supposed to read the archlinux news to know if your next update will fuck up everything.

I learned it the hard way... Once, I had to update a server to get a more recent version of a package.. which ended up deleting all rc.d in favor of using systemd. No warning, no nothing. The only answer I could get is "Why are you updating without reading the archlinux news". Fun couple all nighters time.

Both problems would be caught if you had some kind of testing server (even a VM) and had updated it first. Frankly, you're wrongly blaming the tool.

All problems (or almost all of them) can be caught by using better testing.. That doesn't mean that some tools aren't better than others.

On my production servers, I don't care about not being on the very edge of all technology. I'd prefer something highly secure and stable where all the code can run safely. I also want a very strong community with outstanding documentation. So, all in all, for my use cases, I think going with Ubuntu is a smarter choice. It doesn't mean that I don't use archlinux every day on my own machines where I can have fun screwing it up and hacking it back.

And, just for the notice, that were only small examples, but I'm talking more generally about backward compatibility and expected behavior. A very good example is jquery.. I'm not worried about getting the last update. It's not like ".click" would stop working. Contrast that to others less mature technology (such as express for node.js) where it can be pretty scary to update packages. You better have a gigantic test suites to upgrade without fear of breaking something.

> But how in hell can't you make it possible to use map(print, ..) in python2.x?

`from __future__ import print_function` lets you do that in >= 2.6.

I really love Python, it is my language of choice, bread and butter, and I've been using it since 2005. However, this backwards compatibility break is probably the worst thing that happened to the language, as I think it's creating more project stalling and work in total than is worth any improvements specifically stemming from backwards incompatible changes. It could be because python3 still has not become my target, but that's kind of the point. When is it ever going to be?

When 2.7 stops getting bug/security fixes? Current plan is 2015: http://www.python.org/dev/peps/pep-0373.

What benefits does upgrading from python 2.x to 3 have? Why would you undertake that task? I don't understand why, unless python 3 offers some extremely compelling feature, or you are one of the small percentage of developers who delivers importable python libraries as your main developmental output.

I'm unclear how this is better than doing, as is mentioned in the first paragraph, a single code base that runs in both 2.7 and 3.x versions?

Or is this a case where 2.6 needs to be supported?

You dont. Just support python 3. If you HAVE to support python 2. Then use a symbolic link.

Reply on news.ycombinator.com