Hacker News Reader

Comment by epistasis | Hacker News Reader

About every other month I have to help a biologist who has pulled down an older program from Github debug why it broke in their data only to find that they were trying to run in Python 3 instead of Python 2. These are people that can write shell scripts, but don't program. It makes me aware of all the old code out there that is not used super frequently, but still solves people's problems, and would continue to if there hadn't been breaking changes.

As I write Python 3 now, I am less than impressed with the reasons for breaking compatibility. I'm told that it was necessary for proper bug-free Unicode code, but there's enough difficulty when using, say, gzip, and little enough documentation about the "proper" way to handle Unicode that I doubt that it will actually reduce Unicode-related bugs.

Python is still a great tool, but Python 3 really took the shine off for me.

View on HN

I struggle with this whenever I think about it. On one hand I agree with all of the points you make. But on the other hand, it didn't have to be this way. Python 3 has been out for 12 years now. If people hadn't dragged their feet on converting the mess wouldn't be nearly as large as it wound up being.

IMHO 12 years is a very short time! Of the user-facing programs I use (mostly CLI), the vast majority of the code That gets executed was written far longer ago than that.

Writing some code and not have it executable a dozen years later is a terrible terrible situation. And it wouldn't be this way if Python 2 wasn't EOL.

I'm going to always need to keep a copy of Python 2 around. Always. And if I'm going to change old, working code, I'm probably going to rewrite it in a new language.

As with my previous post I'm torn on topics like this as again I agree with everything stated. And yet, Python was relatively niche in the early aughts so had only really been in the mainstream for a handful of years before 3.0 came out. What %age of live python code in the wild was written after 3.0 was released? I'd wager that's a very large number.

But yet I get why things are the way they are. Heck, it was only last year that I switched to using Python 3. I use it so rarely that it was just easier to stick with the Python 2 which was already installed on my system.

I am not sure why the length of time matters.

They were spending the time doing other important things. They were making the rational decision to work on something that was important and WASN'T already working. The python 2 code was working fine, it didn't have bugs and it didn't need any new features. Other code either didn't exist and needed to be written or had bugs that needed to be fixed.

It makes sense that changing working code never made it high enough on the priority list to get worked on.

On the other hand, my company still has 25 year old C code that still compiles and works under the latest GCC, so...

And on the third hand companies I've worked with had 25-year-old C code bases that seemed to compile and mostly work, but mostly only with -O1 or lower, because they've had UB since the beginning of time but only more recent compiler optimizations reveal the underlying issues.

And this old code is also written in a style (usually global variables and mixed concerns and years of sub-ideal bug fixes and patches) which makes it very hard to change or modernize (forget adding threads with all of this global mutable state).

The only point being that old code may superficially work, but in many cases (in my experience) and regardless of language, it all has technical debt and needs work to keep it relevant.

Does that matter? The code performs its duty and has for 25 years. If there are bugs in it they clearly haven't been important enough to be fixed. Technical debt? Maybe, but if noone is collecting the fees, who cares?

If the situation changes and the original code suddenly becomes a basis for further development then that's a different thing. But cleaning up the code without clear benefit is a waste of resources.

Hats off to original developers. Making code that serves its purpose for 25 years is quite an achievement.

The code gets bug fixes occasionally because things change (scaling past previous limits, changes in the environment, etc) and of course new features are required once in a while. So the debt usually can't be swept under any carpets.

All code rots. If you are lucky to never need to fix it or change it, that's great, but that is (in my experience) extremely rare over time spans measured in decades.

Sometimes a code base runs for decades not because it was particularly robust or well-written, but because it is too painful to replace it due to its complexity or the long ago loss of the knowledge needed to work on it.

Most of our C code actually doesn't compile and work, unmodified, under the latest GCC. There are many new warnings, and like any professional C shop, we build with -Werror (or at least don't ignore the warnings). Since I help with maintaining the build infrastructure at work but I don't own any of the C code, I've proposed setting the new warnings to -Wno-error, but I've gotten pushback from the code owners, saying that they expect the new warnings to catch actual bugs and they want to deal with the fixes.

Sure, there is some amount of C code from 25 years ago that works unchanged, but there's also a good bit of Python code from 25 years ago that works unchanged. (I just tried a few of the examples in the Python 1.3 tarball, released in 1995, and it seems like most of them will work with very minor changes.)

If you compile with -Werror, then you are intentionally running it in no-backwards-compatibility mode. That is not really comparable with Python situation.

Actually GCC is very careful about adding new warnings to Wall because of backwards compatibility. That's a reason why Wall doesn't actually contain all warnings.

Kind of, yes, but it's usually warning us about things that were actual bugs and reliance on unspecified/underspecified behavior that might be changing with newer optimizations.

Not really in my experience. I maintain some relatively old codebases and the most common warnings for old code with newer GCC generally are:

- Warnings recommending the use of parens in code like `if (a || b && c)`. The original code is (probably) not broken, it's just that GCC thinks that it's error-prone to write conditionals that way (I agree). There's also a similar warning for code like `while (a = foo())` where it asks you to add parens to make sure that you didn't use = instead of == by mistake.

- Warnings for unmarked fall-throughs in `switch`. Again, not broken if the fall-through was intentional. That's a fairly new warning, so a lot of all code will trigger it.

- Warnings for some const to non-const pointer conversions (in particular for literal strings in C++). There you could argue that the original code is bogus but as long as the pointer is not written to it's fine. Const-correctness wasn't really a thing in C and C++ for a long time, a lot of legacy code plays fast and loose with const qualifiers, but that doesn't necessarily mean that the code is broken.

- Warnings about unused variables, in particular variables being written to but not read. As compiler get smarter they tend to catch more of those.

Unfortunately, 30 year old Fortran code did not have such good luck in my company. VB6 compiler is hard to come by these days, and I have been tasked with rewriting it for "modern computers" ! The newest executable we have is from ~2008. Kudos to Microsoft to keep it running at least.

But 30 year old Fortran code should compile and run with no problems, unless it used something non-standard, shouldn’t it? In which case, the problem is not the evolving Fortran standard, but the use of non-standard compiler features.

“The Fortran standards committee generally refuses to break backward compatibility when Fortran is updated. This is a good thing (take that, Python), and code written decades ago can still be compiled fine today.” -- http://degenerateconic.com/backward-compatibility/

I don't buy that.

Glibc and GCC introduce incompatible changes all the time.

Not source-incompatible. Yes, if you have a binary linked against an old dynamic libc then you may have to give an alternative LD_LIBRARY_PATH or link it statically after the fact.

Python users are not C users. C users typically understand compiler flags and build tooling.

Some Python users understand compiler flags, but there's also a big community of scientific developers who just want a working language.

Python 2 -> 3 created a lot of problems for them without making their lives easier or more productive.

Perhaps Python 3 should have been called something entirely different, and P2 could have been handed over to a new set of maintainers who weren't going to EOL it.

I've run 20 year old Linux binaries.

Err, that's because x86 architecture is backwards compatible; not because the language those binaries were originally written in is.

Well, if you compile python 2 code using Cython you also can run it in Python 3.

But seriously:

- C never did such change, it doesn't make as much sense for it to do it since it is low level - C has a type static typing, so if they did, they could make compiler refuse to compile until types are right. Python has dynamic typing so these kinds of things show up when running, you can use something like mypy, but unlike C it is optional.

BTW: I saw people using Go as an example that there was no need to add unicode support like in python, and simply store everything as UTF-8. Those people are forgetting that Go already has clear distinction between text and bytes (via string, bytes, runes types), how the language stores text is just an implementation detail, since normally there shouldn't there be access to the internal representation of the text anyway.

The problem is why should people be "converting the mess" in the first place. Backward compatibility, especially when it comes to programming languages, should be a given. If anything, this is reason that people should stop using python.

Is there really that much choice for alternatives though? What else is installed everywhere, has a repl, huge standard library and tons of learning resources, no boilerplate to get going. I don't really like python and still consistently find uses for it.

Ruby meets most of those criteria. It also had a more graceful (though not without its pains) Unicode transition, as it allowed you to mark file by file the status of string literals.

Ruby is a comical example, because they did breaking changes in minor version (I think it was 1.8 to 1.9?).

The thing I think they did better though, is they didn't give people 10 years to fix their sh*t. You either moved on or you stayed with an unsupported version, and people did move on.

For Python 3 the major shifts were in 2015 and 2020. The first year was when PSF announced they won't add any new features to 2.7, and that's when libraries started adding Python 3 support. And 2020 was EOL for Python 2.7.

It doesn't matter how much time you give people to migrate, they will always do it at the very last minute.

The amount of breaking changes was also much smaller than in Py3k.

Also, 1.8 marked the creation of complete compliance test suite (as Ruby 1.8 became an ISO Standard), and that test suite has been expanded since then allowing for alternative implementations.

Because the test suite is your target, not a hard to interpret 2kLOC spaghetti code C function.

> Ruby is a comical example, because they did breaking changes in minor version (I think it was 1.8 to 1.9?).

Ruby was then using a version numbering scheme similar to the old Linux version numbering scheme, where odd minor numbers were development releases and even minor numbers were stable releases. In that scheme a odd minor version is equivalent to a SemVer major release. (Ruby switched to SemVer-ish versions with 2.x)

perl5 meets that except for the REPL. Maybe elisp? I feel like golang could take the place of a lot of Python out there, but not all, certainly.

It would be nice to have an alternative to Python that took backwards compatibility seriously. Might be easier to start from Python2 than to go from scratch. You might be able to add a REPL to perl5, but its syntax isn't the most beginner friendly I've ever seen.

If you replace "installed everywhere" with "reasonably easy to install", there are interesting alternatives among the Lisp dialects. Especially as Python has become a lot more similar to Lisp with many of the additions in Python3.

Things these languages do have in common with Python3 are:

- support for keyword arguments and optional arguments

- support for objects and interfaces (but not necessarily for inheritance)

- handling of names in scopes and name spaces

- closures and lambdas

- list comprehensions

- arbitrarily long integers, complex numbers, a numeric tower

- an empty sequence or empty string is logically false (with a few nuances)

- low-level bit operations (often down to a very low level such as popcount in C)

- easy to call into C code (or Java code if it runs on the JVM)

- full unicode support

- conditional expressions like (a if boolval else b)

- type annotations

- string formatting forms a mini-language

- print is a function

- hexadecimal, octal, and binary literals

- tools for iteration and first-class functions, like partial application

- support for OOP

In addition to this, the members of the Lisp family usually have as well:

- strong support for functional programming

- full and efficient garbage collection

- persistent data structures

- compile to machine code (native code generation or JIT compilation)

- a powerful macro system

- capability to define custom control constructs

- support for pattern matching to members of compound objects, similar to

  (a, b) = name[:2]

- often very good support for parallelism

What exactly would be suited best depends IMO on the use case.

I think Racket is ideal for easy learning, a platform-independent GUI, and comprehensive documentation, and is also useful fir science applications. It can also interact well with C libraries (or, more generally native libraries with C interface), and it is good for scripting.

Clojure is ideal for server applications with a lot of concurrency and a high performance. Its concurrency support is brilliant and best-in-class, much better than Python.

Common Lisp / SBCL is somewhat under-hyped in comparison to Clojure, yet it has an excellent performance, integrates well with C libraries, has a highly interactive repl, a comprehensive library system (quicklisp), a standard build system (asdf), comprehensive documentation for beginners (common Lisp cookbook), it is very usable for scripting, has standard pthreads-like concurrency primitives, a powerful object system, and then some more.

Categorically demanding complete backward compatibility puts huge limitations on the development of the language. I agree that the barrier should be high, and I don't think many disagree that the Python transition was a mess, but for languages to evolve there needs to at least be some way to fix issues that are considered design errors.

There's no optimal or blameless solution here but I think in particular when it comes to the scientist use-case mentioned above, an entire decade is enough to address what is likely a relatively small codebase, it's not just a fault of the language, but also neglect or lack of resources in fields where programming is secondary in nature. (also btw detrimental to the quality of science, as recent cases have shown where programming errors basically invalidated entire studies)

>Categorically demanding complete backward compatibility puts huge limitations on the development of the language

Not really. Just introduce an optional version declaration at the top of the file, which guarantees the old behavior. Then feel free to tinker with the design as much as you want. It's a solved problem and it works well. Sure, you still break people's code if they don't put a version declaration in - but now the fix is a single line.

The fatal flaw in Python's approach was cleaving apart the Python2 and Python3 worlds, making it impossible to mutually import modules. Python2 and Python3 are practically speaking different languages - different syntax, different semantics, different libraries, and a different userbase. As far as the computer is concerned, they are separate entities, with the feeble "2to3" program being the only thing linking them.

Actually I did some small tests, and you could mix python2 and python3 code together through use of Cython.

You can use Cython to compile your python 2 code into .so objects and then include them in Python 3. Then you can progressively migrate code file by file to Python 3. I didn't do it on a larger project, so maybe there are some gotchas, but it seems to work as far as I could test it.

Regarding the differences in languages Python 2.7 was actually a bridge to migrate to Python 3. That's why all features of 2.7 were backported from 3 (this was also the reason people saying: "why should I migrate to 3, it doesn't offere anything new". It didn't because everything new was backported.

Anyway thanks to this it was actually possible to write code that worked on both Python version. I did it several times. It was actually the easiest when you wrote the code for Python 3 and only used features that 2 had. Add all the `__future__` imports and remaining parts filled with the `six` module.

The Cython translation is really clever.

Yes. "from __future__ import ..." could have had "from __past__ import ..." or some other such 1-line mechanism to maintain compatibility. Before the switch to P3, I argued vehemently with (some of) The Powers that Be that doing this was unnecessary and would break the community. The argument was, "Don't worry, P3 is so much better, people will want to change over." Well, yeah, maybe. Personally, I ignored P3 until 3.6 arrived. THAT was enough of a jump to abandon P2.

Similar story here, 3.5 was first version that was even practical to really consider (they kept on making the compat story a bit better because they FINALLY listened to folks). I really wish you and others had more luck earlier, the transition did not need to be nearly as painful.

> Python2 and Python3 are practically speaking different languages - different syntax, different semantics, different libraries, and a different userbase.

Yes, and tools and infractstructure should account for that: Python 2 and Python 3 should use different file extensions, and libraries written for python 2 should not just adapt to Python 3 and break backward compatibility, but come _at least_ as a new major version - better with a different library name, as Go lang requires.

And still it would have been possible to support Unicode in python 3 without breaking backwards compatibility. It was possible to support in Common Lisp which was defined as a ANSI standard in 1994, and Common Lisp implementations like SBCL or Schemes compile to far faster code than C Python (while providing many of its flow control affordances, like list comprehensions, or a numeric tower). Why should it not be possible for Python? Apart from the developers not wanting to do it?

> it's not just a fault of the language, but also neglect or lack of resources in fields where programming is secondary in nature.

A lack of resources for porting and maintaining code to new language versions is totally typical for most of science. There are simply no jobs or incentives to do that, especially if you want to have any kind of career in science.

Further, python developers seemed to think that the bulk of code and maintenance work is in the language implementation and that the many small libraries and scripts are not important. But this is not the case, similar to that in trees, a rather large part of the biomass is not in the trunk, but in the roots and leaves. And the small application scripts and libraries used in science change much more slowly than the main language revisions.

You could argue that this does not matter because most of python's applications are not in science, at least not if you look at the respective amount of money moved. But this fails to take into account that a lot of python's success was so far based on its use in science, especially numerical python and data analysis.

> If anything, this is reason that people should stop using python.

It's also the reason I never started in the first place. It wasn't in any way clear which the right option was during the period of schism, and it went on for an awfully long time.

With that being said I'm starting to tinker with Python now simply because it has a great ecosystem for NLP and ML. I certainly need the former, although how much of the latter I'll need (possibly none) is unclear.

Regardless, it seems to be the de facto standard language for anything that falls under the broad category of data science to the point where for me to pick up a different language would feel very much like swimming upstream.

I'd like to follow this to its logical conclusion: Are you suggesting that it's desirable to make a new language vice improving an old one?

(Related, but separate:) Would you have preferred if Python 3 was released under a different name that didn't include "python"?

No, they should have made the breaking changes more slowly, piece by piece, with a couple minor versions depreciation cycle. Python 3 was too many changes simultaneously. But it also coincided with dramatic changes in the size and variety of the programming community. It was hard to anticipate the troubles, of course easy in hindsight.

Yes, if you're going to break compatibility to a great enough extent. It would eliminate an immense amount of confusion. See what Perl did with Raku.

You fail to understand that they purposely broke compatibility for no benefit. That is not improving the language - that is being stupid.

In python 2 you could specify a unicode literal with u''. So a natural evolutionary option if you had a unicode addiction and wanted python 3 to be unicode native, would be to treat u'' as unicode literals in python 3 as well.

This let's the code run under python 2 still, and then you get unicode in 3. And if you are 3 only, just "blah blah" works as well - nice.

But they didn't do that. They purposely took u'' out to make it hard to support 2 and 3. That is absolute stupidity. In other words, they BROKE compatibility for no benefit - but just to break it.

They did the same thing with byte handling, jumped the io abstraction in - these things all destroyed performance and use cases.

Python is FOSS. If there was a big enough market for python 2 it would have been forked and maintained by a separate entity. That it didn't happen proves this is overblown

There are entities out there with support contracts that stipulate Python 2 support for a while. Red Hat is one such entity[1].

[1] https://access.redhat.com/solutions/4455511

I guess there isn't really a propper need for a perpetuation of python27. This is mostly people who don't really see a difference between the versions other than that their print statements suddenly fail or that unmaintained dependency they used in that code half a decade ago complains. Who cares about the security pitfalls involved with using outdated python when tour code doesn't work?

Of course python could have dealt better with backwards compatibility, but the python code I wrote was transfered to version 3 pretty quickly.

> If there was a big enough market for python 2 it would have been forked and maintained by a separate entity.

Perhaps some chose instead move to other languages.

It happened multiple times, and one of the times, the project needed to be renamed, causing a huge argument, lots of eyes on the project, and momentum to slow. There's a huge social cost to a fork.

https://github.com/naftaliharris/tauthon/issues/47

The dependency web makes it really hard to upgrade one part. You have to upgrade or rewrite all your dependencies and then do the same to your code: I tried this several times in the ‘07-‘12 timeframe and gave up each time: the cost wasn’t worth the benefits.

Ultimately, I stopped writing Python: these days I write Common Lisp for personal projects and Clojure/JavaScript for everything else.

It was made a lot harder by their efforts around intentional breakage. You couldn't do import from past (ie, to get print "" working). You couldn't do u'' to get unicode compatible string handling between both versions.

When I was first starting, I almost gave up on Python because the print "Hello world" script that I copy-pasted from an online tutorial failed to run.

The hardest part (for me) of getting started with a new language is setting up the environment to run properly... Installing libraries, configuring environment variables, pointing the IDE to the interpreter, etc. So of course that's where I assumed my mistake was.

It was only much later that I realized that I was running Python 3 (newer is better right?) and the tutorial was written when Python 2 was the latest version. So the print statements changed.

And of course, "print" is in everyone's first program. So it broke the experience for all first time users.

> So it broke the experience for all first time users.

To be fair, Python 3 provides a very clear error message if you try to use print as a statement, although I don't know when exactly that was added:

    $ echo 'print "Hello World"' > test.py && python3 test.py
      File "test.py", line 1
        print "Hello World"
                          ^
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Hello World")?

>To be fair, Python 3 provides a very clear error message

To be fair, nobody reads error or warning messages.

then those people are helpless? you can lead a horse to water.

I agree with the parent actually. My first python program was a json read and print something. 4 lines, maybe 5. I had a need to parse json and print something specific and I suspected I would be able to do it in python. After 4 hours, I was able to and I felt happy. This is not to complain about that 4 hours.

But when I think back at that 4 hours and how much time will it take to do it today, it reminds me of all the learnings. Is it load or loads? Which one is from file? What is a json object? Do I need to quote? If so how? Why does it work with the small block but not the big block? Is it the nesting? Am I doing something wrong?

I think the same is true of hello world. Hello world is a newbie presumably who has not written a single line of code. Well you copy and paste and it does not run. Yes the error is there but hey I copied working code : did I install something wrong? Should I check some website? It says it is python but is it? You can't imagine the questions that will come to a newbie. We need to be mindful of this rather than have a stack overflow mindset.

This is largely due to conditioning of over 30+ years of bad error messages.

  The program exited with error code -2
  Error MS1337 occurred
  Something went wrong. Contact your Administrator

Ideally; error codes should provide information about how to fix the problem (which python in this case does). That falls on the developer side of things (and may not be possible in many cases).

Learning that error messages are useful and can solve your problem, and de-training the reflexive "This message won't help me", lies on the end user side of things (assuming the devs did their part).

> Ideally; error codes should provide information about how to fix the problem (which python in this case does). That falls on the developer side of things (and may not be possible in many cases).

Rust generally tries to do that, although the fixes it proposes tend to lack context, and can thus lead you into the wrong direction entirely (e.g. once upon a time it would tell you to go and implement a trait on your type when the issue was that you'd forgotten a trait bound on a generic type, it still kinda says that but now there's a snippet which adds the trait bound so it's less error-inducing).

Python itself has a habit of contributing to this long history of extremely hostile messages.

If run in interactive mode, the Python interpreter takes its cues from vi and deliberately frustrates the obvious methods to quit. Ctrl-C results in Python whining KeyboardInterrupt but doing nothing helpful. Typing 'exit' or 'quit' both result in messages that make it clear that the interpreter knows what you want to do but won't comply out of spite.

More fun can be had with certain versions of Python 3 when running on Windows, where the Python print function would throw an exception if told to print Unicode characters. This made printing to the console non-portable because Unicode characters had to be stripped if running under Windows. This seems to have been fixed, however.

Still existing are the misleading warnings that Python produces if you forget to include the self parameter on a class method. Python emits an error that blames code that calls the method for passing one too many parameters instead of identifying the underlying problem. Python also gets similarly unhappy, and points fingers at the wrong code, if the brackets are omitted when instantiating a class.

Since Python is dynamically typed, these errors can't be identified until execution. Requiring all-paths testing to catch syntax errors is a special kind of user hostility in and of itself.

Python is an utterly horrible language and it's deeply unfortunate that it has become so popular when so many less error-prone, less obnoxious, alternatives exist.

> Typing 'exit' or 'quit' both result in messages that make it clear that the interpreter knows what you want to do but won't comply out of spite.

The interpreter knows no such thing. Both quit and exit are regular python objects that implement a __repr__ method, both of which return a string that inform the user how to exit. The __repr__ method could have implemented exit, but that would have been poor form (and create issues with tooling), set bad precedent, and go against python coding guidelines.

Ctrl-C giving a keyboard exception rather than exiting is a good thing, as it is far more likely that you are trying to terminate a long running statement, or exit out of sub code block, than trying to kill the interpreter process. This is also the standard behavior of almost any interpreter you care to name (irb, node, clisp, shell). This is because the interpreters try to mimic unix shell semantics (CTRL-D also kills your shell), since they are, after all, shells.

Since Python 3 can detect this case and accurately determine the user intent, it could have just executed the statement as if it were written with parentheses. Had they done that alone, they would have significantly decreased the number of short Python scripts that broke during the transition. Or they could have made a --permissive flag that would do this.

That removes the nudge. People should update their materials.

If you think that, do you think Word should start refusing to open .doc documents at some point?

Code is sometimes alive and changing, and sometimes an artifact. Requiring that code become alive after being an artifact carries significant risks. One might say "code should never be unmaintained" but I think that ignores the reality of limited resources that all of us deal with.

Well, we can either have continuous innovation and improvement with proper security, or we can have long-term support for code that is unmaintained, but it currently seems impossible to have both. I don’t think anyone would consider the .doc format a paragon of design elegance or security.

I'm willing to bet latest Word will have issues opening earliest Word documents ?

It is phenomenally good: https://news.ycombinator.com/item?id=21451713

Or people can just read the error and fix their code.

Having print as a statement was a mistake with a lot of downsides (see https://www.python.org/dev/peps/pep-3105/ for the rationale).

It would take quite a lot of effort to offer the new print function and the old print statement at the same time, and it would require either a lot of special cases or renaming the new print function.

And there's really no upside. If the only legacy thing in your script is the use of print statements, that's trivial to fix by hand or with one of the automatic converters.

> And there's really no upside.

Yes there is, the print function is the biggest syntactic incompatibility. Besides the (optional) u marker for unicode strings that they reintroduced later. Most of the other difference (like changed packages) are semantic.

You would have been able to import and run Python 2 in Python 3, maybe with a couple of "if"s or some magic in the module loader. Breaking compatibility like this felt petty and deliberate and made it really hard to have one codebase target both python 2 and 3 (temporarily).

But as the GP mentioned, it can be fixed automatically.

In this specific case I think it's a non-issue. If python3 was limited (or limited except in very niche cases) to changes that could be made programmatically, we wouldn't be in this situation. Everyone would just run 2to3.py and never look back.

It's also extremely easy to fix.

Python is a teaching language. One of its strengths is that it can be your first language and still be one you use professionally a decade later.

A "trivial" obstacle to a seasoned programmer can be an insurmountable obstacle to a beginner.

The problem here is that print "statement" can't be upgraded to a log "function" easily as the programmer grows in experience.

A print "function" conversely is quite easy to upgrade to a log "function".

In addition, "print" is almost always a code smell in professional code. The moment you need to localize, change destination, etc., "print" goes right out the window.

I have lots of beefs with Python 3. Changing print to a function is NOT one of them.

That's a terrible message for people trying to run other people's code, unfortunately. It wouod be more useful to say "Are you trying to run Python 2 code in Python 3?", If Python detects enough "2isms" in the file.

Except that is imprecise and can be misleading. It is certain you called print incorrectly. It is not known if you just made a mistake or are trying to execute python 2 code.

This was a while back. In version 3.5 (5 versions after the change was made!) the message was just:

SyntaxError: Missing parentheses in call to 'print'.

This doesn't give the "did you mean" hint, and while it's perfectly understandable for someone who's gotten as far as learning exceptions in Python, that usually comes a few chapters after "hello world". It's definitely enough to throw a newbie off. And even if they're a veteran of other languages, they might have been exposed to a wide variety of error messages like this that commonly indicate a problem elsewhere, and having never yet successfully run a python script at this point, the prior for an error in configuration is much higher than for the tutorial being wrong.

Oh man I had the exact same experience about 10 years ago. I couldn’t get “hello world” to work despite doing everything the tutorial said. Disheartened me from trying any more programming for several years.

> little enough documentation about the "proper" way to handle Unicode that I doubt that it will actually reduce Unicode-related bugs

I won't claim here that the compatibility break was actually worth it, but I am sure that it reduces Unicode-related bugs.

It's easy to write Python 2 code that seems to work fine but immediately crashes when it has to process Unicode. But if your Python 3 code works at all it'll probably handle Unicode fine, or at least handle it better than naive Python 2 code would.

Whenever I have to write Python 2 code that has to handle Unicode, I mainly just try to write it so that it would be able to run in Python 3.

> It's easy to write Python 2 code that seems to work fine but immediately crashes when it has to process Unicode. But if your Python 3 code works at all it'll probably handle Unicode fine, or at least handle it better than naive Python 2 code would.

Yeah, that's the biggest benefit, and the reason python 3 handles it better, is because it crashes early. When it it sees that you are making an incompatible conversion, it refuses to run without waiting for some specific unicode character to show it.

> Whenever I have to write Python 2 code that has to handle Unicode, I mainly just try to write it so that it would be able to run in Python 3.

Thankfully I no longer have to write python 3 code, but from my experience, it works the best to write the code for python 3, then adapting it (sometimes with help of six package and imports from `__future__`) to work with 2.7. It is easier than do it the other way.

Agreed completely.

There were so many routes forward that didn't involve breaking everything but still would have allowed fewer unicode bugs (maybe). And they could have done a lot more to make the transition easier if they wanted to - some of it was just in your face type breakage to a prove a point without considering some underlying use cases (a lot of internet protocols, headers etc still ASCII).

What is nuts to think about - if tweaking some unicode handling is worth creating an incompatible version, what happens when some random python developer has another itch to scratch that they consider so important it is worth break a language over. The idea that these types of fixes require breaking compatibility is absolutely insane if you want to address a larger / longer term market.

Complain all you want about Microsoft, C etc. Even go of all languages (because google has an otherwise terrible rep in this area!!). But some of these languages and companies really do seem to be able to deliver or intend to deliver longer term compatibility.

When businesses and lifetimes are measured in decades (not months) it is VERY nice to build on stuff that has longer support horizons.

Python 2.7 was supported for 9+ years, python 3 was released in 2008, with development beginning before then. You had almost 15 years of warning from initial announcement to eol. Python3 predates golang.

> And they could have done a lot more to make the transition easier if they wanted to - some of it was just in your face type breakage to a prove a point without considering some underlying use case

Such as?

> (a lot of internet protocols headers etc still ASCII).

What do you mean here? Py3 correctly handles http headers encoded as ascii bytes. Yes the decision to make http headers ascii bytes and not basically anything else is bad in hindsight, but that ships sailed.

>You had almost 15 years of warning from initial announcement to eol.

Why should I have to do labor to fix things that already work?

> Why should I have to do labor to fix things that already work?

You don't right? Python 2 still works.

Maintaining python2 and python3 forever in perpetuity wouldn't be free either. Why should they provide free labor in perpetuity so that you need no provide any?

In theory if there was sufficient need the entire community of python2 users could collectively fund said support but I don't see how they can demand it for free.

You don't have to it's not like PSF made Python 2.7 stop working on January 1st 2020.

You don't need to, provided you're willing to also maintain a Python 2.x fork on your own instead, or willing to pay someone else to do it for you, or willing to risk no more patches to the interpreter. You do you, boo.

Even a simple message in an exception statement, "you are running Python 3; could this be Python 2 code?"

I claim that this would have negative EV. Most exceptions aren't due to version mismatch. You could add a line for a very specific set of syntax errors, but those aren't very useful.

Also having your new version suggest using the old version is counterproductive.

Maybe... But there are two correct solutions (using an old version or upgrading the code) and such message would communicate clearly the need to decide between them.

I think you're misunderstanding what was meant by "longer term compatibility". The issue isn't that the migration period was too short, it's that there was a such a big discontinuity in the compatibility in the first place.

Python 3 provided no extra value at a high migration cost and painful library ecosystem issues. This is in contrast to typical programming language version migration, where breakages are either avoided entirely or are extremely localized. For example C++11 had some breaking changes compared to C++98. I don't know if I ever noticed them, but it didn't matter since C++11 added so much value that everyone wanted to migrated ASAP, and anyway there was no need to move the whole ecosystem at once.

No matter how long the transition period was for Python 3, there was not going to be a good time to migrate. So most people would leave it until the very last minute, resent every hour of time spent on it, and sometimes put the option of rewriting in another language on the table.

> Python 3 provided no extra value at a high migration cost and painful library ecosystem issue

I disagree with this. There was a lot of value add for specific (and relatively common) workloads. If you were doing unix automation you didn't gain much (but I argue this is because unix does unicode bad, not because python). But web development did, even if most of it wasn't obvious or was too soon to be obvious.

As for c++ the effort the language goes to to maintain backwards compatibility has costs, both in terms of syntax and semantics. Those costs push people to rust etc. Much the same that migration to py3 pushes some people to go. It's difficult to say which is right without the ability to time travel.

We have line of business code (not python) that still runs in production that was written over 20 years ago.

Hell Larges parts of Windows is still code that is decades old...

This "well you had plenty of time" shows just how much many modern / web developers do not understand Enterprise code, code that goes into manufacturing or other long running processes where "new" machines are 15 years old

I'm not clear what your point is. Much of the python3 code I use today was written 10 years ago (or more) and only barely had to be touched to work on 3.

> other long running processes where "new" machines are 15 years old

Then...pin to a 15 year old version and use that. No one is forcing you to upgrade.

Web developers re-write their entire stack all the time. In a business with tons and tons (and tons) of little apps and services and command line tools etc, NO ONE wants to (or can) go through all this stuff to update it because someone decided the way unicode should be handled should change. The value gain to those 100's or little apps and utils is essentially zero. I do think a LOT of people got a lot better at making sure to lock configs down - it used to be you could basically ship the python file and as long as you were not doing something weird the system python / distro python would work fine. Now you really need to version pin everything, because python 3 was a nightmare for all the pip libraries and the knock on effects just showed up everywhere.

Take a read here. Py3 was supposed to better - but in a LOT of use cases this better turned out to be "miserably slower". So you were asked to do a lot of work to get your code to run slower.

File handling in the new unicode world was something like 2-3x SLOWER. This is not percent, something which took 3 minutes all of a sudden took NINE minutes because of the io module.

URL parsers became a nightmare - dual implementations of codecs started poping up.

https://lucumr.pocoo.org/2014/1/5/unicode-in-2-and-3/

For a "better" python, I didn't really start using it until 3.5 or so. Now it's much more pleasant, but at the key point of transition when everyone was going to check out the new thing, it was slow, put lots of extra bumps that appeared unnecessary to solve the problems that needed solving etc.

I'm not sure what your point is. You still had 12+ years of warning. You didn't need to migrate in 2010. I'd guess that the majority of application code migrated in the past 2 years, but that doesn't mean that you didn't have time to plan, and prepare.

> File handling in the new unicode world was something like 2-3x SLOWER. This is not percent, something which took 3 minutes all of a sudden took NINE minutes because of the io module.

Yes, unicode is slower. If you weren't using unicode you could and can still do all your processing in bytes mode.

That's not the way it worked. You couldn't migrate your code until your libraries had migrated. They didn't always want to migrate because their userbase was generally mostly on 2.

This is what worries me - the lack of understanding of how broken these backward compat breaking changes are. Mercurial, built on python, took 10 years of work to get to python 3 - for basically no reward (they just need bytes and unicode is not super relevant to them) as just one example of MANY.

Let me give you an example. in Python 2 you could use u'' for a unicode string. Perfect, I could update python 2 code to use this or b'' if I needed bytes so in the python 3 setting the right things could happen. The python developers literally TOOK THIS AWAY in python 3. So now, for no reason we can ever hope to understand, you couldn't write code that work work in 2 and 3 by using u'' and b''. This was the type of screw the developer in your face attitude that developers got.

Let's be clear, library folks would need to write code that worked for python 2 and 3 for a long time. And instead of making this easy or at least reasonable, the python developers went out of their way to basically screw folks over here.

Others will remember that the very reasonable formating approaches to bytes (yes, those are useful) was trashed in python 3, again for no seemingly good reason. % operator .format etc. WHY? I'm serious.

"You had 12 years of warning" - hey how about python follows some other languages. Make the update compelling so folks are excited to move. Think about things that will make it easier instead of just pulling all these silly stunts. You couldn't practically move initially. Something as simple as doing u'' in python 2 that would alias properly in 3 - was a no go.

This is what worries me about posts like yours, python developers have really no clue how stupid this type of backward compat breaking approach is if you have limited upside and go out of your way to make it hard to support both versions. It is critical that lesson be learned - but it seems hopeless.

Anyways, go and others I think are going to make a reasonable run at this sort of utility language space with I think some stronger efforts around backward compatability. I think the next backwards incompatible change in python (python 4?) will probably see a much bigger migration off python.

> They didn't always want to migrate because their userbase was generally mostly on 2.

Yes, this doesn't change my argument. You had 12 years of warning. You couldn't migrate immediately, but you had 12 years of warning to plan your migration.

> I think the next backwards incompatible change in python (python 4?) will probably see a much bigger migration off python.

You aren't paying much attention to the python ecosystem then. I wouldn't expect a python3-like breaking change for another 10 years, if ever.

> Mercurial, built on python, took 10 years of work to get to python 3 - for basically no reward (they just need bytes and unicode is not super relevant to them) as just one example of MANY.

Mercurial took ~4.5 years to migrate, from April 2015 to November 2019. Mercurial also took a roundabout way of doing the migration that likely made it take longer (https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journ...)

> The python developers literally TOOK THIS AWAY in python 3.

And then readded in python 3.3, around 7 years ago.

> . You couldn't practically move initially. Something as simple as doing u'' in python 2 that would alias properly in 3 - was a no go.

I agree you couldn't migrate immediately! But you still had 12+ years of warning! I'll repeat myself: I expect most application code was migratedn in the last 2 years. But the maintainers of those applications had 10+ years to prepare for the migration of their application. This prep took many forms. For some it was nothing, for some it was helping dependencies migrate to 2/3 compat. For some it was planning the eventual migration. But every single one had more than a decade of warning.

I think Perl 5 (despite the much bigger Perl 6 fork) used a language pragma for unicode. When Java introduced generics, they had to use type erasure.

It does seem like if you put in some effort and do some decidedly unaesthetic things, you can bolt on and backport a lot. No developer dreams of maintaining dirty hacks at scale, though.

Please note that Perl 6 did not start out as a fork, but over the years effectively was considered to be one. Which is why it changed its name to Raku (https://raku.org using the #rakulang tag on social media).

> When businesses and lifetimes are measured in decades (not months) it is VERY nice to build on stuff that has longer support horizons.

Indeed. I'm in the process of writing some simple web stuff. I considered python, but decided against it because I'm doing this for free, and don't want to be asked to come back and fix it in a decade because "Python 4.6 needed a feature, and it'll break compatibility with Python 3"

Yes, they could still just run it with Python 3, but do I really want to go through the effort of explaining how to configure that instead of just handing over a working binary?

Why would you worry about that for Python? The community learned their lesson. I'd worry more about other language maintainers who haven't had the learning experience.

Because with Python the possibility is still there. With C/Rust/Go, it's simple to statically compile and remove the possibility of that type of issue.

It's not a program that will be running on a server, so I'm not really concerned about security fixes/updates to libraries.

Uh, just use the old interpreter. Same situation.

And then the distro they are using stops shipping Python 3 by default because "No one should be using that anymore.".

While I could bundle python with the program, that's getting to be more hassle then it really seems like it's worth. In addition, I'm not particularly comfortable with configuring python, so there's a non negligible chance I'd screw that up and be in the same position.

It's not that it's impossible for Python to work in this situation, it's that me doing it is a bad idea. I'm sorry if my original comment implied otherwise.

Internally, Python 3 still doesn't use Unicode, and it's pretty arbitrary whether a specific unicode sign is allowed as a variable name.

What you're talking about Have you tried to run VB6 code in latest VB?

Things are getting broken all the time, the major mistake IMO what PSF did is giving 10 years to migrate code. Everyone waited to middle of 2019 when they actually started to migrate and the 10 years only allowed to created FUD.

I used Python for a while years ago and asked a core developer about this, he said they’re not paid for support of that nature and to talk to Red Hat. For them it’s just a fun project, no matter who (foolishly?) built a product on it. They’ve made it out to look like a toy language, but it’s taken seriously.

So I went with C# instead. I’d choose it again over Python.

Did you end up talking to Red hat? I'm curious about how much "support" they'll provide. I can't imagine anything other than back porting security fixes, but even then it seems like they'd be pretty limited.

I wonder if there's any merit to python3 kicking off a "is this script python2?" on some subset of common errors that crop up.

Then it could tack on a "looks like you're trying to run a python 2 script on python 3" final error to stderr.

That’s a great idea. Perhaps even offer to 2to3...

The whole Python 2 vs 3 thing turned me off when the split first happened when it was “Python 3000” back in, what, 2008 or so. Every time I‘ve had to do something with Python since then there has inevitably been some sort of incompatibility issue that popped up, although it’s been quite a lot better these past few years.

It always left a bitter taste in my mouth and I always feel like I’m fighting with Python every time a project comes up where I have to use it. There are so many other languages that are a pleasure to use and I’ve always wanted to like Python, I just feel like I’m cursed to always find Python needlessly annoying.

Personally, I don't mind some inconvenience if language designers occasionally need to make breaking changes to allow improvements, and I've had to port enough scientific Fortran code to newer languages to limit my sympathy for porting from Python 2 to 3. Computer languages aren't static, and sometimes I need to do some work to keep up as they evolve.

I bet the old Fortran versions are supported and compile decades after the 'newer' languages stop being supported...

I have compiled Fortran programs from the 70s on modern platforms without changing a line. The compiler, OS, and CPU architecture had all disappeared but the programs still worked correctly.

I've 'modernised' such an old program then come back a few years later, realised the 'modern' ones don't run anymore and reverted to the old Fortran version.

That is likely a compiler issue. Besides being modern, Fortran 2018, 2008, 2003 have no backward incompatibility issues.

You misunderstand. I 'modernized' to Python (similar to the GP). Then the Python version stopped being supported so I reverted to Fortran..

And Python 2 scripts continue working just fine with the Python 2 interpreter.

Which is no longer supported.

Well, it is open source. I'm not expecting Python 2 to get completely dropped from Linux distros in the near future. (I could be wrong; I will be disappointed if I am.)

There was an article the other day about problems mixing different python versions on Mac. I admittedly didn't read much of it—but I was somewhat baffled by the problem. I have Python 3 installed on my mac as "python3", and Python 2 (which comes with macOS) as just "python". I think that's the right way to treat them—as different, independent interpreters which can co-exist, as do, say, Python and Ruby.

OK but the latest Fortran specification and compilers explicitly support all Fortran77 code and a really large amount of F66 as well.

Will there be Python 3.8 interpreters in 2100? F77 will still compile and run better than it did in the 1980s.

There were minor breaking changes in Python 2.4, 2.5 and so on. So Python certainly had the gradual minor breaking evolution down pat.

As if all those shell scripts are portable across time and distributions. Or PHP code. Or really any other code. How many builds of your software are truly reproducible?

Python 2/3 incompatibility is always magnified for some reason, but it's nothing out of the ordinary in my experience.

Didn't we just recently have a highly upvoted/heavily debated HN submission on semver? Enough said...

> As I write Python 3 now, I am less than impressed with the reasons for breaking compatibility. I'm told that it was necessary for proper bug-free Unicode code, but there's enough difficulty when using, say, gzip, and little enough documentation about the "proper" way to handle Unicode that I doubt that it will actually reduce Unicode-related bugs.

This was absolutely essential. Python 2 tried to fix by introducing unicode type. It was hacky, and because they looked for a shortcut, they did a very bad thing by introducing implicit casting (so python 2 automatically converted between str and unicode). They did implicit casting because they didn't made the effort of fixing the stdlib to properly handle unicode.

At that point not only there was a problem that str represent two different things (bytes and text), but also authors that were using unicode (to make their code futureproof) make their life actually harder (in python2 even when you know how unicode works, implementing it correctly is a minefield, it is so easy to do it incorrectly and end up with code that works, but blow in special circumstances).

Python 3 basically decided to rip the bandaid off and do it correctly. And that's what happened. When you program in python 3 you no longer have to think much about unicode. To convert between byes you need to explicitly convert from text to bytes (.encode()) and bytes to text (.decode()) but this is not a concern unless you are doing some kind of serialization.

While the breaking unfortunately was necessary (a lot of python 2 code, while it runs it is broken), I'm wondering if it was possible to do it a bit more developer friendly.

I think things would be much easier if instead of changing the behavior of `str` type, Guido would introduce a `text` type (along with `bytes` type), then later deprecate it. I think that would help a lot. Although in that case modifying stdlib would be unavoidable, so it would be a lot of work for them.

BTW: MyPy was also a tool that could help with the migration, but once again they IMO dropped a ball. Basically when MyPy is used in Python 2 mode, it follows Python 2 typing system, which means it still allows for implicit casting. I think what was really needed is to highlight these issues to allow for fixing python 2 code before doing the migration. The way mypy worked required you to first migrate your code to Python 3 and only after that you could start fixing the text/bytes issues.

> These are people that can write shell scripts, but don't program.

Being able to figure out that you're using a different major version of Python is definetly expected of today's biologist. This is part of the profession now.

> Python is still a great tool, but Python 3 really took the shine off for me.

People forget that part of the reason why the Python maintainers were so snotty about Python 2 to Python 3 was because they went through this before.

Anyone remember the gnashing of teeth that occurred when trying to drag people off of Python 1.5.2? I sure do.

I want to throw anybody complaining about backward compatibility in 2020 into Python 1.5.2 because THAT'S what you're advocating for.

> little enough documentation about the "proper" way to handle Unicode that I doubt that it will actually reduce Unicode-related bugs.

There are hundreds if not thousands of pages of unicode documentation. If anything the problem is that there is way too much for any normal person to actually understand it.

For most people though, knowing when to normalize strings to NFC versus NFKC is all they need to not unknowingly accumulate tons of technical debt.

You are right, there's tons of Unicode documentation out there. The problem is how to use encodings in Python. What parts of the standard library are going to be OK with Unicode, and which parts are OK with bytes, and more importantly which parts reject bytes/Unicode?

For example, I coded along in Python 3, not changing and IO handling for the longest time, but when I Tried to use the gzip module I suddenly had to learn a whole bunch more knowledge about Unicode. I looked around for about 10 minutes for a some sort of best practices guide, but ij lieu of that just mucked with encode/decode functions until it worked. I don't know if I did conversions at the right spot, or where the guidelines are for changing between bytes and characters, but it didn't matter for my data at the time and o had a deadline so I couldn't really justify spending another hour at the time to figure it out.

Maybe I'm just trying to justify my behavior to myself, but I think this sort of problem solving is the norm. I tend to try to adopt best practices and educate myself more than the other programmers I have worked with, if not more, and am the one introducing flake8 etc. to code reviews, so I'm not the worst sort of programmer, though I certainly wouldn't claim to be top-tier either.

> What parts of the standard library are going to be OK with Unicode, and which parts are OK with bytes, and more importantly which parts reject bytes/Unicode?

This is coming from someone who's written a lot of Java (Java's had the byte/char distinction since forever); the way Python 2 pretended byte strings are strings really was broken and not what you want. What the changes in Python 3 mean are that you have to actually think about if something is bytes or a string, and you have to be intentional in choosing encodings.

I've recently been using some other languages that use a Python 2-esque "strings are byte-strings that are probably UTF-8" model, and I actually think it's a better choice than what Python 3 went with.

Sure, you can argue that iterating over bytes is less "correct" than iterating over Unicode code-points. But iterating over code-points isn't very useful either -- maybe that code-point represents the letter B, but maybe it represents the third of seven characters and modifiers and zero-width joiners that make up one family emoji.

This is the worst part. 12 years to change the language’s Unicode model and, after all that, the new model turns out to be a bad design because in the meantime, everyone else has standardised on UTF-8.

> I actually think it's a better choice

What should `str.reverse()` do? It's very different for byte strings and UTF-8 strings. Or `str[4]`? `str[1:]` can corrupt a string for one definition of []. What about Windows users? There's a chance Powershell is using Windows-1252.

That's the thing, though -- even in Python 3's model, neither of those operations produce the expected result in all cases. If I reverse "réal", I might get "laér" as expected, or I might get "láer" -- it depends on if é is represented as one single character or as two characters using "latin letter e" + "combining acute accent".

The same thing applies to indexing and slicing. You might get an actual character, or you might get something invalid by slicing halfway into a multi-code-point sequence.

So Python 3's model isn't actually that much better than Python 2's at handling these cases. But it comes at a significant cost, introducing "unicode vs bytes" confusion, and making it more difficult to write correct programs for platforms like Unix where filenames aren't necessarily valid Unicode.

Actually it is. What you suggesting would completely break those operations. Compared just not getting what you want. In any way you can tell python how to behave through normalization of the unicode.

> What you suggesting would completely break those operations. Compared just not getting what you want.

I reversed a string several times in Python 2 and not once did my computer catch fire! I just got a string back that, in certain situations, didn't contain the characters I was expecting. And then my father had to have a talk with me about Unicode, and how computers sometimes use multiple bytes to represent a single character, and so if you rearrange the individual bytes you get different, usually more broken-looking characters.

And when my child runs "🇨🇦"[::-1] in Python 3, I'll still have to give them The Unicode Talk. I just need to use the word "codepoint" instead of the word "byte". The situation is basically the same--if you know enough about Unicode to understand Python 3's behavior on certain non-English text, you know enough to understand Python 2's behavior.

> And when my child runs "🇨🇦"[::-1] in Python 3, I'll still have to give them The Unicode Talk

What were you expecting to get when doing reverse on 🇨🇦? The flag symbol is encoded using regional indicator symbol letters: C, A. When you reverse it you get regional indicator symbol letters A, C which represents Ascension Island. In python2 you get a total garbage.

The flag representation is not part of unicode, unicode only specifies that you can encode country codes using ISO 3166-1 alpha-2 and it is up to the application how to present it.

> What were you expecting to get when doing reverse on 🇨🇦?

My future child sees only one character in that string, an emoji. (If you put the cursor to the right of 🇨🇦 and press backspace, the whole flag goes away, so that must mean it's only one character, right?) If they reverse that string, they'd expect to get the same result back.

> The flag symbol is encoded using regional indicator symbol letters: C, A. When you reverse it you get regional indicator symbol letters A, C which represents Ascension Island.

Right, this is exactly what I'll have to explain! Just like how in UTF-8, "é" is encoded using two bytes: 0xC3 0xA9. So if you reverse that, you get bytes 0xA9 0xC3, which do not represent "é". It's the exact same problem, you've just moved up one level of abstraction. (The Python 3 version of the problem isn't unique to flags -- there are all kinds of modifiers and ZWJ sequences and combining characters that get messed up if you rearrange individual codepoints.)

A few languages have tried to solve this problem in other ways, like how Swift considers each character of a string to be a "grapheme cluster", but that comes with a lot of its own drawbacks.

IMO this isn't really a Python issue though. By unicode standard how to present it is left to the frontend. Even displaying "CA" might be enough.

With combining character sequences there's at least a normalization and you can transform to the right representation based on your need.

As you mentioned that Swift allows one to operate on grapheme clusters, and it comes with advantages but also some issues as you pointed out.

The topic is complex but Python 3 does what unicode standard specifies, while Python 2 would output a garbage in similar scenario.

But Python is a duck-typed language. And most of the time, a bytearray's quack and a chararray's quack sound similar, if not the same [0]. Java can afford to distinguish between bytes and characters because it is strongly typed throughout the language. In Python, whether 2 or 3, as a developer you do not have the luxury of informed reasoning whether something is bytes or a string, because your functions' input parameters can be either.

In hindsight, I think the proper decision would have been to ensure every string has an accompanying encoding attribute, and refuse operations on two strings of different encodings (or even perform on-the-fly encoding changes). At least that wouldn't have created a source-incompatible language.

[0] Here's some of the operations that should absolutely be invalid on a byte array, but aren't because Python3's bytes object isn't actually bytes, it's 8-bit ASCII: lower(), title(), upper(), isalpha(), isdigit(), isprintable(), splitlines()

> Java can afford to distinguish between bytes and characters because it is strongly typed throughout the language. In Python, whether 2 or 3, as a developer you do not have the luxury of informed reasoning whether something is bytes or a string, because your functions' input parameters can be either.

Python is a strongly typed language, and you can check a function's argument types at runtime with isinstance(), and inspect those types' class hierarchies with issubclass().

The parent meant statically typed. Meaning it shows the errors at compile time.

Yes, you can check at runtime, the problem though is that especially on Python 2. Those checks don't happen very often.

Python 3 though will crash if you try to use bytes as text or vice versa. The python 2 though it will do an implicit casting and execute code, which will run and look like it runs correctly until you get an unicode characters, then it will crash.

Java won't even let you compile such program. You could get somewhat similar protection if you annotate types and use mypy to check them.

While I agree with you about some of those, `isprintable` only makes sense in a byte object, it doesn't make sense on text which is printable by definition. Perhaps it should be called isasciipritnable, but yeah.

> In hindsight, I think the proper decision would have been to ensure every string has an accompanying encoding attribute

This wouldn't have made things any more source compatible. Adding more types doesnt address the issue that python2 string was multiple things masquerading as one.

> In Python, whether 2 or 3, as a developer you do not have the luxury of informed reasoning whether something is bytes or a string, because your functions' input parameters can be either.

I guess they can be, but the code might also work on an int tuple, even though the operation doesn't make any sense. Here's a case where duck typing does something, but it doesn't make sense.

    import functools
    def sum2(vals):
        return functools.reduce(lambda a, b: a + b, vals)

Python's built-in sum won't work on strings or lists of strings (so much for saying the language is duck typed), but this will. It's also weird because addition is associative, but not with strings, so changing a+b to b+a changes the behavior and gives you a very cute answer to the "reverse a string" algorithm:

    reverse = sum2

In practice, every legitimate use I've seen for duck typing might as well use what would be an interface in Java. I see duck typing as interfaces without the formality of declaring them. If you violate the implicit contract, funny things happen, and this is why Typescript got so popular and Python added type annotations.

On a similar note, both Java and C# made an early and mostly incorrect decision to support utf-16 in the native string and char types. This choice also made some programmers to believe that every unicode character fits in one char, causing bugs, etc.

Yet, I have not seen a rush to "fix" any of them the Python way.

You are mixing the internal representation of text with the change to make distinction between text and bytes.

The python incompatibility wasn't really the Unicode it was that older versions of python mixed text with bytes, and because of that a lot of python 2 code is actually broken. It might run fine, but as soon as input are Unicode characters then many python 2 applications crash.

Python 2 tried to implement Unicode and implemented an `unicode` type. But it actually made things worse due to them adding implicit casting.

The proper way to fix the issue was actually clearly making the distinction what is text and what are bytes, something that was in both Java, C# (both of them are younger than Python BTW, the Unicode was in works when Python was released).

he UTF-16 used by Java and C# as I said is just an internal representation of the text, and that's actually trivially changed since normally those things are not accessible for developers. The only way where it could somewhat leak is default encoding, and maybe names of the methods from stdlib. But those things are really easy to change without much breakage.

Side note, Python internally stores text in many different ways if the text is only out bytes it uses traditional C-string (for performance). Otherwise it also stores as UTF-8, UTF-16, UTF-32 based on characters and requested conversion. So those conversion you said Java or C# did Python 3 does all the time (based on need) without breaking running code.

I don't think I am mixing it up. The char type is not a unicode character just like the byte is not. Yet, these are the characters in a string type. Python went to fix this, jvm and clr didn't.

I'm not that familiar with C#[1], but in Java char is absolutely an unicode character. That's why you have char and byte types. Java clearly separates them, and so is python since version 3.

[1] although from quick googling it looks like C# handles things the same way as Java.

This used to be the case when every unicode character could fit in 16 bits. Now, most methods that take char may fail: https://docs.oracle.com/javase/7/docs/api/java/lang/Characte...

As somebody who programs a lot of python for roughly a decade now I really like Python 3 (although I tone quite a while to switch over completely). Unicode handling really got much more straightforward (which you might notice only if you need to deal with Unicode, which many people in the anglosphere don't do all that often). I like the new formatting options, type annotations all great stuff.

And quite frankly most python 27 code I wrote worked after adding brackets to the print statements.

https://www.python.org/dev/peps/pep-0394/

so “helpful”!

You should never break backwards compatibility in a language. It's its undoing.

Yet. https://www.jetbrains.com/lp/python-developers-survey-2019/#... https://www.jetbrains.com/research/python-developers-survey-...

Production code must be maintained. I've seen many shops with legacy 2.7 code that is stuck due to breaking compatibility in 3.x. Put it this way, if my production code suddenly needs to be patched due to non-security changes then that has to come out of somebody's budget. What else could we be doing with that budget than patching code from 2.7 to 3.8. How do you justify that to your budget director?

"We can either do this now, slowly and carefully, or do it quickly and haphazardly when some dependency has a cve only patched in 3.x"

As long as you are willing to spend some of that budget continuously patching and maintaining the old 2.x interpreter to address security vulns as they are discovered (IF they are discovered), sure. No need to upgrade!

If you haven't had the resources to port a codebase to a new language version over the course of twelve years, you clearly haven't had the resources to maintain it.

> What else could we be doing with that budget than patching code from 2.7 to 3.8. How do you justify that to your budget director?

Can't speak for smaller businesses, but in larger companies, there is so much "waste" that the cost in $$$ to do this is insignificant in comparison. And if you choose to spread this effort out over, say, 5 years the annual cost is miniscule.

Not only is this untrue in the case of Python, I think it's untrue in the case of basically every other successful language:

- ANSI C was incompatible with pre-ANSI C. Current C has various minor incompatibilities like the removal of gets().

- bash defaults to non-backwards-compatible behavior and lets you set various shell options like "compat31" if you want it.

- Newer version of the JDK generally require code fixes.

- PHP has made various backwards-incompatible changes, like the removal of magic quotes in 5.4.

There's also many more cases of languages getting forked in backwards-incompatible ways and going on to be successful. C++ is doing fine despite not being fully backwards-compatible with C; VB.NET broke compatibility with VB; zsh isn't POSIX-compatible by default; there were a million variants of BASIC back in the day, and still are a million variants of SQL; etc.

Reply on news.ycombinator.com