Hacker News Reader

Comment by jwilk | Hacker News Reader

> 2to3 is all you need 95% of the time

This is absolutely not the case in my experience.

2to3 tends to produce something that appears to work at first glance, but it actually doesn't.

2to3 gets all the good low hanging fruit syntatic fixes I've found, but it really comes down to the str/bytes divergence and how mucked up the mixing was in the py2 code. His comment here is the clinch... "pdbparse is among the worst offendants I’ve seen since, as any parsing library, it will happily handle and transform bytes-like data as string."

This has not been accurate at all in my experience. It really came down to whether the code had supported non-ASCII text before — if there's a huge amount of bytes/str conflation, it's a mess but if it followed good Python 2 practice it's been very close to convert and run.

This lines up with my experience. Strangely I had someone submit a bunch of slightly passive-aggressive "Py3 support" pull requests to a couple of my repos which were basically the result of running 2to3 on the code. They hadn't even bothered to run the unit tests (which obviously failed spectacularly). It did however prompt me to get Py3 support working properly which was actually a good thing.

> Strangely I had someone submit a bunch of slightly passive-aggressive "Py3 support" pull requests

Is there a more polite way to go about this? I submitted some "Py3 support"-style pull requests to a repo out of the blue a while back. In that case it was just a few minor changes to some fairly small scripts and I did those by hand, preserving Py2 compatibility. But I did wonder at the time whether it wasn't a bit jarring for the owner to see a stranger pop up with a bunch of pull requests like that.

> Is there a more polite way to go about this?

This is the usual process for submitting a large change to a project:

1. Check the issue tracker or mailing list for an existing discussion about Python 3 support. If there is one, read the discussion. If there isn't one, start the discussion. Mention that you're willing to make the changes if there is interest. Remember that supporting both Python 2 and Python 3 places a burden on all future contributors.

2. Keep the changes minimal and easy to review.

3. Review the changes yourself before sending a patch/pull request.

4. Run the test suite.

5. Test your changes in all versions of Python the software supports.

Forgetting #1 is usually the cause of tension when people submit large patches. The person who submits a patch wants to see their patch included. The authors of the project want to make sure it aligns with the goals of their project and doesn't break their software.

You have any examples? I was planning to use 2to3 in a project, can you explain in which case it doesn't work? Thanks!

First, I'd recommend python-future rather than 2to3, it has additional fixers and replaces some with better versions.

Second, those only do the fairly straightforward changes, fixers are simple AST transformations they don't do complex type analysis or anything, you should use them as a starting point but don't expect the work to be over by then: the 2->3 transitions has a number of API and semantics changes which these tools can't handle, text model changes are the biggest one[0] but they're not the only one by far.

If the project is non-trivial, having a good test coverage is absolutely crucial.

Basically, 2to3 handles the first 90% of the work which are mostly drudgery and syntactic fixes, you're still on the hook for the second 90% which is more subtle.

Source: finalising the conversion of a ~200k SLOC codebase to cross-version compatibility.

[0] not just the strict separation of text and bytes, but APIs being set to one or the other so you have places where you want to ensure bytes, others where you want to ensure text and yet others where you need "native strings", some APIs (csv) are also incompatibly altered.

Lots of people mention strings, but for a concreteish example: you have some code which reads bytes from the network, say "cookie", and uses that as a key into a dict. Later you try to access the stored value with headers["cookie"].

This works in python 2. It silently fails in python 3 because b"cookie" and "cookie" are different keys. Not really related to whether the original code supported Unicode or not.

I agree, although different people's experience may differ simply because they were or were not using certain language features often.

What bites me all the time is the division operator. The 2to3 code will run, but produce incorrect results. String / byte handling is an obvious other one.

Reply on news.ycombinator.com