Python adopts a 12-month release cycle | Hacker News Reader

Python adopts a 12-month release cycle

lwn.net

231 pointsTomte170 comments

From my point of view, the Python release cycle presents several challenges in terms of two-way communication with users.

Ideally, we want people to download the beta releases and give feedback on their real world experience with new features while there is still time to make changes. (That is almost the entire point of a beta-release accompanied by a feature freeze).

Actual practice tends to fall short of the ideal for several reasons:

* Not many users download the beta and try out the features. On the bug tracker, we tend to get very little feedback on our beta releases.

* Some features land with little to no documentation (the walrus operator, for example). This makes it difficult for users to assess the feature.

* The PEP process is decisive. Once a PEP is approved, it is too late for developers to give feedback. The feature will happen regardless of user feedback.

* Downstream tooling needs time to update, so it is difficult to try out a beta when you can't get your dependencies to run (MyPy, Black, requests, etc).

>The PEP process is decisive. Once a PEP is approved, it is too late for developers to give feedback. The feature will happen regardless of user feedback.

That shouldn't be the case. There should be a PEP-re-evaluation stage, once the feature is available.

It might not affect major parts of the PEP (that should be handled in the original PEP discussions), but for things that are found to be problematic in actual use, once the feature is available in beta, changes should be considered before the final release.

A new PEP can be created. At some point the RFC phase ends and the feature needs to be committed to.

Generally speaking, for a pep to be approved, there has to be a working proof of concept.

This isn't always true, but I've seen it be the case for most recent peps.

That isn't practical. The PEP process is already exhausting and we aim to solicit feedback before the PEP is decided. By the time a beta is out, we're already in feature freeze and it is somewhat late in the game.

>That isn't practical. The PEP process is already exhausting and we aim to solicit feedback before the PEP is decided. By the time a beta is out, we're already in feature freeze and it is somewhat late in the game.

Isn't it the case tho, that a PEP is usually just a document + discussion? I've seen someone having a PoC sometimes, but iirc it's not often readily available for use.

Unless a feature is available to try (in alpha, beta, whatever), people can't really evaluate its ergonomics and how it affects or integrates with their use cases.

I know that at least a few languages, including Go and Rust, don't commit to a feature after the relevant proposal is accepted, but have version you can try, that they might change and/or revoke.

As someone in the Python community often says, "there must be a better way"!

Maybe a PEP could include the stage where there is something like a "from __pep_proposals__ import new_feature".

This might slow down the PEP rate (if every new candidate feature needs a trial implementation), but that's not necessarily a bad thing. Or it could just be required for larger changes to the language.

(THE RaymondH? !!!)

On the beta testers front I think it's not impossible to get a dozen or more full time devs at a major bank I work at to commit to downloading and using it and giving feedback - can you indicate what might be useful ?

This would be a big help. The process is simple. Download the beta, try-out the features, and let us know how they work out in real world code -- is it learnable or prone to misunderstanding, are the docs intelligible, does it solve the problem it was intended to solve, does it fit in with the rest of the ecosystem, etc.

APIs are really hard to change once released. Prior to release, we can fix just about anything.

I'll see what I can get going next week. cheers nice idea.

For me the walrus operator is an example of how things can take a turn for the worse. It's a bad idea. I have yet to see any example that justifies its existence. Put a different way, I have yet to see a reason for not extending "=" to add this behavior. This is how a nice language can be ruined and can become a Tower of Babel scenario. In some ways this is how J is a ruined version of APL, because it completely destroys what the language was about in the first place. Next thing we know we are going to end-up with 23 different assignment operators, one for each chromosome pair.

The other part I don't get is the religious insistence against not implementing something like a "switch" statement. This is one of the most useful constructs in all of computer science and Python rejects it due to what amounts to religious belief. In fact, one of the most powerful tools you get when moving from assembler to C is the "switch()" statement. If you've ever coded and maintained the equivalent in assembler you know exactly what I am talking about.

Because, you know, a pile of "if" statements is somehow better. Ah, OK, with "if" you can use data types other than integer. Sure, because we couldn't define a Python "switch" to work with anything other than integers...like a "switch" that could take a dictionary and route based on key/value matching, etc.

Every single state machine I have written in who knows how many decades --in C, C++, Objective-C, PHP, Verilog, etc.-- has had, at it's core, a single or multi-layer "switch" statement. Can't do that in Python, 'cause.

Could you give a concrete example of how the walrus operator is worse for the language? I know you've admitted that this is a rant but without anything to back it up, it isn't much interesting to read.

As for the switch statement, I'm inclined to agree although if we see `switch` as syntactic sugar for a bunch of other conditional statements, then what makes it a more important candidate for inclusion than the walrus operator?

Thanks in advance, I'm genuinely curious to know.

The walrus operator has very clear problems once you start to use the assigned variable at the same line it receives a new value. That's not really relevant on practice, since nobody does it, but it moves a language from a philosophy of simple semantics into one where people are ok with undefined behavior or a pile of exceptional cases. This change does smell really bad for people that have seen it before, so those tend to overreact and start complaining that the sky is falling.

Specifically on the case of Python, it used to promise simple semantics a long while ago. This changed at some point, and the semantics were already complex before the walrus operator. With the operator added, semantics becomes a bit more complex, but this is not a main feature of the language anymore, so it's not a big change.

Yet, pattern matching would have a very small semantic footprint, and would solve the same kinds of problems (despite having a very different nature), but was very vocally rejected without any of the deep analysis that went into the walrus operator. It looks like this happened because they would have a very large syntactic footprint, basically completely changing the appearance of the language, but the process left the people that focused on semantics lost, without understanding what was happening.

> Could you give a concrete example of how the walrus operator is worse for the language?

Because it isn't necessary? All you have to do is extend the syntax to now allow the one-and-only assignment operator to function in other places, for example, in an "if" or "for" statement and you are done. Anything else can be taken care of with parenthesis.

For more examples of how I don't see any applicable ambiguity, see my reply below to a different question:

https://news.ycombinator.com/item?id=21426133

I mean, to me this is like adding a wing to the back of a four door family car. It's useless complexity.

We want to reduce complexity, not increase it.

BTW, I read through the entire PEP [0] and still find no way to answer this basic question in favor of the walrus operator:

What's wrong with simply extending the functionality of the existing assignment operator?

I looked through the examples and mentally made the change. I can't see any need for the new operator. Case in point, this is one of the examples given in the PEP:

    Current:

    reductor = dispatch_table.get(cls)
    if reductor:
        rv = reductor(x)
    else:
        reductor = getattr(x, "__reduce_ex__", None)
        if reductor:
            rv = reductor(4)
        else:
            reductor = getattr(x, "__reduce__", None)
            if reductor:
                rv = reductor()
            else:
                raise Error(
                    "un(deep)copyable object of type %s" % cls)



    Improved:

    if reductor := dispatch_table.get(cls):
        rv = reductor(x)
    elif reductor := getattr(x, "__reduce_ex__", None):
        rv = reductor(4)
    elif reductor := getattr(x, "__reduce__", None):
        rv = reductor()
    else:
        raise Error("un(deep)copyable object of type %s" % cls)
  


    My version, after declaring that "=" can now be used this way:

    if reductor = dispatch_table.get(cls):
        rv = reductor(x)
    elif reductor = getattr(x, "__reduce_ex__", None):
        rv = reductor(4)
    elif reductor = getattr(x, "__reduce__", None):
        rv = reductor()
    else:
        raise Error("un(deep)copyable object of type %s" % cls)

Why do we absolutely need the walrus operator again?

And yet I go back to the "Current" pre-walrus code and have to ask: What is wrong with it? It's clear, very clear. And, nothing whatsoever in this logic is changing at the microprocessor level because of the use of the walrus operator. Even with we do not extend "=" to be able to use it in extended form, there is nothing whatsoever wrong with the original code. This basic structure has been used for decades in myriad languages. Not sure why there's a need to reinvent a wheel for basically zero gain. Extend "=" so it works in a few places, sure, that might clean-up code, but walrus? C'mon.

To add to this, let's take a look at the simplest portion of the above and think about where this ultimately ends-up: As code being executed by a microprocessor; machine language.

    reductor = dispatch_table.get(cls)
    if reductor:
        rv = reductor(x)

versus:

    if reductor := dispatch_table.get(cls):
        rv = reductor(x)

versus (the case of extending "="):

    if reductor = dispatch_table.get(cls):
        rv = reductor(x)

Now let's look at rough machine-level pseudo code.

In the first case:

    - Load address of "cls" into appropriate register
    - Jump to address where "dispatch_table.get()" code is located
    - Upon return, the result, a memory address (pointer) is stored in a register
    - Load the memory address (pointer) to where the "reductor" variable's data is located
    - Store the address that came back from "dispatch_table.get()" at that location
    - Compare this and jump if null
    - If not null we execute the code that follows

Great. What would the pseudocode for the second case (walrus) look like:

    - Load address of "cls" into appropriate register
    - Jump to address where "dispatch_table.get()" code is located
    - Upon return, the result, a memory address (pointer) is stored in a register
    - Load the memory address (pointer) to where the "reductor" variable's data is located
    - Store the address that came back from "dispatch_table.get()" at that location
    - Compare this and jump if null
    - If not null we execute the code that follows

Hmmm, I see a pattern here. What would it look like if we extended the functionality of "=" instead?

    - Load address of "cls" into appropriate register
    - Jump to address where "dispatch_table.get()" code is located
    - Upon return, the result, a memory address (pointer) is stored in a register
    - Load the memory address (pointer) to where the "reductor" variable's data is located
    - Store the address that came back from "dispatch_table.get()" at that location
    - Compare this and jump if null
    - If not null we execute the code that follows

OK. Well, there's the point. If the code has to do exactly the same thing it will ultimately end-up executing exactly the same code on the processor. If it executes any more code than the old approach it will not, by definition, add any functionality, therefore, it is a waste of time. If it does exactly the same thing we should then question why it is necessary in the first place.

These two are syntactically equal and in Python there's no way a linter can distinguish between these two:

    if reductor = dispatch_table.get(cls):
    if reductor == dispatch_table.get(cls):

A human being can only distinguish them through careful inspection. The walrus operator not only prevents that problem, but makes the intent unambiguous.

> Not sure why there's a need to reinvent a wheel for basically zero gain.

What's being reinvented? No one claimed this was an original idea. It's just inline assignment with a syntactic tweak to prevent a well known problem.

Using = for assignment expressions introduces ambiguity in if statements: is

    if (x = 3):

Intended, or not?

There is no ambiguity. This example is invalid syntax.

In Python yes, but the post I responded to asked why they settled on a different assignment syntax for expressions.

It was to avoid that ambiguity (or error prone pattern, if you prefer) which is present in, say, C.

C/C++ compilers (at least, all the ones I know about) squawk about it unless you add an additional set of parentheses around the assignment.

Throw a warning? C compilers require additional parentheses of you want to use the value of assignment as a conditional. It's impossible to make a mistake there unless you're dumb on purpose that is disabling or ignoring compiler warnings. That being said I think it matters little if it's a separate operator or not. It solves a real issue I'm the language caused by assignment not being an expression. Better later than never I guess.

I don't understand, what is wrong with extending "=" such that in the example you gave "if(x=3)" it stores 3 in x and then makes the comparison?

More accurately, in Python it would be:

    if x = 3: # x is assigned 3 and then tested
        do_something_here()

This is different from:

    if x == 3: # x is compared to 3
        do_something_here()

Very clear difference between the two statements.

What if we want to assign and then compare?

    if (x = 3) == y:
        do_something_here()

or...

    if x = 3; x == y: #Python evaluates left to right...
        do_something_here()

In fact, you could...

    if x = 3; x == y; x = x + 1: #Python evaluates left to right...
        do_something_here()

Not super elegant, but it makes sense. I wouldn't want to write code like that in any language unless it was very well justified.

What if we want to compare and then assign?

    if x == y; x = 3: #Python evaluates left to right...
        do_something_here()

Side note: Why, oh why, don't we have pre and post increment/decrement (++/--) in Python? Another religious decision.

I guess I am having trouble seeing where this ambiguity might be. What is ambiguous about usage that is currently not legal and is later defined as a new way to make assignments?

In other words, nobody is using it that way today because it doesn't work. Tomorrow we say: From now on, you can make assignments inside of several conditional statements.

What's ambiguous about that?

This isn't foreign at all, is it? I mean, we have been able to do this in C for decades:

    int i;
    for(i=1; i<=3; i++)
    {
        printf("%d\n", i);
    }

In fact, you can do this:

    int i;
    for(i=1; i<=3; i=i+1)
    {
        printf("%d\n", i);
    }

That's TWO assignments, not just one. No special operator or syntax. Last I checked no airplanes have crashed, MRI machines stopped working and nobody died because of this. Forgive me but, I just don't see the ambiguity at all. Once you define the extension of functionality the rest is not a problem.

That said, I am eager to learn. I've only been designing hardware and software since about 1982 or so and have programmed in nearly every mainstream language in existence on platforms ranging from embedded to Silicon Graphics supercomputers and even the web. It is quite possible I am confused or just don't understand something. I do have a very pragmatic view of this stuff. By which I mean to say: Simple is usually better and there has to be a very, very good reason to reinvent any wheel.

> I don't understand, what is wrong with extending "=" such that in the example you gave "if(x=3)" it stores 3 in x and then makes the comparison?

There are many theoretical and quasi-religious arguments you can make about this point, but I think the real reason is a lot simpler: in practice, allowing assignment during expression evaluation has led to tons and tons of bugs.

The number of times a programmer accidentally uses a single "=" sign instead of a "==" in a conditional statement compared to any efficiency gains of avoiding an extra line of code, has empirically resulted in a net negative in productivity.

This isn't a theoretical or philosophical point, it's one rooted in collective experience. This "collective" experience may not match your own, but honestly, for experienced developers it's such a minor issue, that I'm fine giving up 0.001% of my productivity if it allows less experienced programmers to gain more than that.

I don't believe this. Sorry. It really isn't that complicated. If someone intends to compare and, instead, assigns by mistake, well, that code isn't going to do what they wanted it to do in the first place.

Yet, at a more fundamental level, there's nothing wrong with code like this (taken from the PEP):

    reductor = dispatch_table.get(cls)
    if reductor:
        rv = reductor(x)
    else:
        reductor = getattr(x, "__reduce_ex__", None)
        if reductor:
            rv = reductor(4)
        else:
            reductor = getattr(x, "__reduce__", None)
            if reductor:
                rv = reductor()
            else:
                raise Error(
                    "un(deep)copyable object of type %s" % cls)

What is wrong with this code that desperately needed fixing through the introduction of a new operator?

There's something in the PEP about programmers wanting to save lines of code. Really? The code at the microprocessor level is the same. Heck, if lines of code were so important I would still be programming in APL, where I could shrink a hundred line C program into just a few characters on a single line of code.

I guess my problem is with the idea of fixing something that isn't broken while ignoring stuff that is just nonsensical. A simple example of this is the lack of pre and post increment/decrement (++/--), frankly, it's just silly. We are writing "x = x + 1" instead. The real irony is that every microprocessor I know has increment and decrement instructions! I mean, it's hilarious! "x = x + 1" is likely to be compiled to such an increment instruction, which, is the machine language equivalent of "++" (in rough strokes). One could not make this stuff up. Who are we fooling?

> I don't believe this. Sorry.

Your belief conflicts with Swift, which also disallows assignment in if conditions for this reason [0]. With GCC, that warns about "suggest parentheses around assignment used as truth value" when compiling with `-Wall` [1]. With the most popular Javascript linter [2].

The tricky part is that "that code isn't going to do what they wanted" isn't always immediately obvious. For example, when I google'd "comparison assignment typo", this is one of the first results: https://gitlab.freedesktop.org/wayland/weston/commit/209e8f1... If the author was expecting that test to pass with the ==, then = would introduce a subtle but devastating bug. In the future, someone could break that function but it would never get caught by the unit tests.

[0] https://docs.swift.org/swift-book/LanguageGuide/BasicOperato...

[1] https://gcc.gnu.org/ml/gcc/1998-07/msg00085.html

[2] https://eslint.org/docs/rules/no-cond-assign

> There's something in the PEP about programmers wanting to save lines of code. Really? The code at the microprocessor level is the same.

No, they don't want to save lines of code, they want to save time to understand code. Disallowing assignments during expression is one way to advance that.

But I'm not really understanding your bigger point here. Who cares about such minutia? Is "x++" versus "x += 1" versus "x = x + 1" really all that different to the programmer? When have you proudly used the shorter versions and actually made code better in some marketable way?

These programmer purity metrics are silly and counter-productive, we should instead focus on programmer quality and speed to get things done.

BTW, you have also hit upon why APL uses distinct symbols.

When your assignment operator is "←" you are never going to confuse assignment for comparison.

    Assignment:

    a ← 23

    Comparison:

    a == 23

Extending this to something like Python:

    # Assignment
    if x ← 3:
        do_something()


    # Comparison
    if x == 3:
        do_something()

In fact, you could argue that once you have a dedicated assignment operator you no longer need the "==" construct, because "=" is enough, which is precisely how it works in APL. The above examples the, would become:

    # Assignment
    if x ← 3: # x is assigned 3 and then evaluated
        do_something()


    # Comparison
    if x = 3: # Is x equal to 3? It even reads exactly as it is meant
        do_something()

So, if we want to reduce error and this kind of ambiguity the solution isn't to now have two different assignment operators using easy to mistype characters but rather to have distinct symbols that cannot be confused. Hence the genius of APL, where the notation is, as Iverson put it, a tool for though and the abomination of J, where notation was tossed out the window in favor of ASCII soup.

The ambiguity could be resolved easily by adopting notation that makes sense. Why do we use "A*B" for multiplication when we can use a proper multiplication symbol "A×B"? Or division? "A÷B" Not to mention the logical comparisons; Or:"A∨B", And: "A∧B" , Nor: "A⍱B", Nand: "A⍲B" (these are real symbols used in Logic, BTW, as well as APL).

In a world with several good examples of pattern matching implemented in modern high-level languages, a C-style switch statement should probably be shunned going forward.

The problem with this idea is very simple: code bloat.

Modern software engineers have very little connection to low level code; what actually happens at the processor level.

If you are even in a position to interview someone for a software dev position, try this:

- Have them write a small program do to something simple; find a sub-string in a long string

- Now have them take their program and convert it to pseudo assembly language, explain what goes on in memory, where, how, etc. In other words, it doesn't have to be real assembly from a real processor, yet it has to resemble how things actually work.

From my experience most applicants flunk this test. They have no clue whatsoever.

And so, we produce tools, languages, libraries, and all manner of "pattern matching in modern high-level languages", they use them without a single clue and the resulting code is bloated and slow, to say the least.

Many years ago I wrote a genetic solver as a part of an iOS app that ran on an, at the time, iPhone 3S. It was an absolute dog. Unusable. You simply could not get results anywhere close to real time. An iPhone 3S, in the context of what the code entailed, is a massively fast computing engine. There was only one reason for which this code ran like crap: Objective-C and all the truly unnecessary "modern high-level language" garbage that came with it.

I re-coded the genetic solver in C++ with great care to manage the entire process efficiently. It ran THOUSANDS of times faster. In fact, it ran in real time (in the context of what the UI had to do) to the point where there was no lag and the user could interact with it as needed.

A lot of this stuff comes at great cost. The problem is modern programmers don't have a clue.

I agree with you in spirit, but, it's Python, so bare metal speed is a lost cause already. You're already supposed to drop down to C for any performance critical code anyway.

In addition, even when people think they know what is going on under the hood, most people are taught how things worked about circa 1992 at best.

> This is one of the most useful constructs in all of computer science and Python rejects it due to what amounts to religious belief.

What utter nonsense.

There was extensive discussion, and it ran into the limitations of Python's semantics. Here's a related problem, a common mistake for new Python coders:

    x = []
    for a in [1, 2, 3]:
        x.append(lambda: a)
    print([f() for f in x])

It prints [3, 3, 3], of course, because people don't realize all the lambdas are closing over the same 'a'. And there's simply no way around this with the way Python's scoping works.

Read PEP 3103. Every possible implementation was potentially badly misleading to users:

1. Building the dispatch at compile time would result in mysteriously immutable values. 2. Semantics would likely be different outside a function. 3. Or it would sometimes use dispatch and sometimes use if-else chains, leading to inconsistent performance. 4. Or they'd have to add completely new syntax for static data.

If they just made it syntactic sugar for an if-else chain, even that's misleading, because people will invariably be surprised to learn it's not doing a simple dispatch.

There's nothing religious going on here, Python is simply constrained by its original design. If various proposals to add new syntax for static data come to fruition, then a switch statement would make perfect sense and have clear semantics, and they will be glad they didn't try to kludge some half-baked crap together like they did with Enum.

> The other part I don't get is the religious insistence against not implementing something like a "switch" statement.

Off the top of my head, something like a switch, but more general and a function rather than a statement:

  def switch(key, opts_dict, *, default):
    return opts_dict.get(key, default)()

or, if key is of a comparable but non-hashable type:

  def switch(key, opts_list, *, default):
    return next(
      (o[1] for o in opts_list if o[0] == key), 
      default
    )()

I can think of other simple variations to do exactly what is wanted in other situations; it's easy enough to code a specific tool for the situation with what Python gives you that, in the situations where a chained set of if/else isn't what is called for, it's trivial to whip up something tailored to the specific need. C doesn't have dicts or iterators or first-class functions in the base language (at least, with convenient syntax), so it needs special syntax to cover a much smaller space than what Python conveniently handles without special syntax dedicated to the problem.

Actually, what you want is not a switch, it's pattern matching

I used perl almost exclusively before moving to python and it is very easy to use regular expressions.

Now I use python and love how it is better at separating the expression from the python language (with perl you were never sure if you had escaped things correctly)

but pyython is just... clumsy

  m = re.match(r'xyz',str)
  if m:
    do a
  else:
    m = re.match(r'wxy',str)
    if m:
      do b
    else:
      m = re.match(r'abc', str)
      if m:
        do c:
      else:
        ...

vs:

  if m := re.match(r'xyz',str):
     do a
  elif m := re.match(r'wxy',str)
     do b
  elif m := re.match(r'abc',str)
     do c
  ...

I was curious how this changed from their previous process. It looks like they've historically averaged about once per 18 months for the past few releases.

    3.4.0 — March 2014
    3.5.0 — Sept. 2015
    3.6.0 — Dec.  2016
    3.7.0 — June  2018
    3.8.0 — Oct.  2019

I'm not familiar with their release process — were those "scheduled" releases, too, but just at 18 months (plus a little slop for release QA)? Or were they just ready when they were ready?

In the PEP 602 document (https://www.python.org/dev/peps/pep-0602/) they state that the previous schedule was roughly 18 months. One reason they wanted to forgo this time frame is that there is pressure to stuff many changes into each release because missing the deadline means waiting another 18 months.

I'm wondering why Python has such an slow release cycle? There are new Node.js releases on a weekly (and sometimes daily) basis.

Why not ship any improvements you have immediately to the users?

I like a more stately release process for a language (compared to applications where I do love frequent releases).

Stability and predictability are important. Porting code to new versions is not something I want to be continuously working on, and increasing the number of OS/language/library combinations to debug is painful.

Once a year sounds about right, with important bug and security fixes being pushed out when needed.

Naive question: does Python follow SemVer?

No. Python versioning pre-dates semver.

That said, point releases (like 3.7.2 vs 3.7.1) are generally considered backwards compatible - they are suppose to "just be" bug fixes and security fixes.

3.x to 3.x+1 has no guarantees about backwards compatibility.

It's also very hard, in any language with "eval", to support new language features and backwards compatibility.

    try:
      eval("async")
    except NameError:
      print("pre-3.7")
    except SyntaxError:
      print("3.7 or later")

Similarly, eval("a@b") gives a NameError on newer Pythons, and SyntaxError on older.

I don't think that has anything to do with `eval`. It's just hard to do with a dynamic language.

And that's not to say it shouldn't be attempted!

"Eval" is one way to implement a dynamic language, so I'm confused by why you seem to write those as very distinct concepts.

Is it correct to interpret your last line as saying that Python should support SemVer?

If so, the only way to get SemVer is for every release to increment new major number, as every single one has backward incompatible behaviors going through eval.

  eval("a:=1") - SyntaxError before 3.8
  eval("async=1") - SyntaxError starting with 3.7
  eval("1_0") - SyntaxError before 3.6
  eval("a@b") - SyntaxError before 3.5, NameError after

It's true there are other ways for a dynamic language to trigger backwards compatibility issues than by going through eval(), but my goal was to give concrete examples of how difficult it would be for Python to make guarantee about backwards compatibility because eval() does exist.

My point is really an aside, but none of your examples actually require eval. Any code could have used `async = 1` directly as a variable, for example. And dynamic languages needn't support eval, though they typically do.

It's just a challenge with permissive languages that expose as much of their internals as Python, but a practical view could still be taken — for example, it could be possible to run all the test-suites of packages in the PyPI on a new version to test for breakages.

You're right; code could have used 'async = 1' before.

Another technique is:

  try:
    import test_module
  except SyntaxError:
    ...
  except NameError:
    ...

Nonetheless, my point wasn't really to show that eval() was essential, only that any language which supports eval() will have these problems.

Is C a permissive language?

C99 added support for BCPL-style comments, causing

   int a = 1 //*  */ 2
                    ;

to give different behavior. C11 removed gets(), and broke code which just happened to define its own aligned_alloc() while also #including stdlib.h.

Those releases are 12 years apart and are definitely not using SemVer.

I’m not at all against breaking changes, nor am I trying to litigate which languages are harder to extend non-breakingly, I’m just surprised that there’s no compatibility guarantees between 3.x releases.

I meant that C example to get some insight on what you mean by "permissive language" and why that was important to the topic.

What language makes the compatibility guarantee you're looking for? I can't think of enough such languages to warrant being surprised about Python's behavior.

How's that different from just writing `async`, without `eval`?

i.e. if the string's static `eval` isn't doing anything?

The module would not be able to be imported due to the syntax error. Eval is required to make the module syntax correct.

That being said, don’t do this. Check the version explicitly.

Maybe releasing more often could improve stability and predictability? Isn't it like they say: if it hurts, do it more often.

This is also what the JavaScript community is moving towards, both for the language specs where new features are "released" once they're ready (see: https://2ality.com/2015/11/tc39-process.html) and their respective implementations in browsers and Node.

As far a I can tell, those release processes (with many millions of users and full backwards compatibility going back to early beginnings of JavaScript) are neither unstable nor unpredictable.

Things to consider. Javascript has a much stronger separation between language specification, standard library, and runtimes than Python. The Javascript standard library is famously sparse, and there are multiple runtimes with significant usage and development. Python on the other hand has a much larger standard library to maintain, and effectively has only one runtime (yes, I love PyPy too, but realistically 99+% of runtimes are CPython). And they are all implemented by the same group.

The mechanics of quickly releasing language level changes, or standard library changes, or runtime changes in Python I feel do not fully map to that of the Javascript experience.

The language usage models are also very different. In the majority of Javascript use cases, I feel that you either fully control the runtime and code (like in a server, or something like Electron), or you have a fairly intelligent server determining what the runtime environment is and providing backwards compatibility shims for you (polyfills on browsers).

Certainly something like the first case exists for Python, but certainly the tooling for deploying onto arbitrary runtime and capabilities doesn't quite exist. Indeed, the Python packaging story is one of its weaknesses right now.

Python certainly could be made into something where the Javascript like process would work, but I'd certainly agree with anyone in the core Python development team who believed that they thought their energies could be focused better on other priorities.

That's a great answer, thanks. I think it's really interesting to see how differently both open source projects are managed and maintained.

From the (very nice) link:

"There will be one release per year and it will contain all features that are finished by a yearly deadline."

That seems to be what is going on here with Python. Most JS implementations have differing amounts of the early stage features they have implemented, and given their open source nature some of the Python implementations can do the same thing.

I feel like I'm missing your point but to me it looks like the languages are converging on broadly similar release processes.

Ah my bad, I think that's the wrong link describing the old process. I remember a recent talk by Axel (author of that post) describing the new process where features are accepted as soon as they are done (i.e. stage 4). But I can't find any articles by him to back this up, so maybe my memory is failing me ;-)

Update: ok, as soon as a proposal reaches stage 4 it's considered finalized and part of the standard. A proposal can reach stage 4 at any time during the year. At that point browser vendors will start implementing those. https://flaviocopes.com/es2019/

I think release frequency and stability are pretty orthoganal. I haven't had a node upgrade break my code since 0.8 (which was obvious a pre-1.0 release). Likewise, Rust releases every 6 weeks and it's rock solid.

I like bug fixes and security updates the day they are ready.

Language updates that change the language I prefer once every 25-40 years.

Because people depend in Python for real work, and are not used to the platform play havoc with them and change under their feet every few months...

This is great if you code in pure Python! The problem is that every new Python version seems to break something. Python 3.8 was released almost two weeks ago, and some popular libraries still don't work with it, e. g. SciPy, Matplotlib, OpenCV, PyQt, PySide2 and probably a bunch of other libraries that depend on the above.

There is work towards a stable abi, that should help with such things, eventually.

Yes, exactly. Every new 3.x release ends up being a not insignificant amount of work for us to update our build system for our not pure Python packages.

Maybe a faster release cycle will help Python and friends insulate from this issue better. Akin to os security updates.

According to the announcement, it should get better: "Now that the stable ABI has been cleaned, extension modules should feel more comfortable targeting the stable ABI which should make supporting newer versions of Python much easier"

That's not because 3.8 broke something. The packages you mentioned require compiling. To make it simpler for users, the maintainers also provide precompiled versions. The issue here is that no one precompiled them for 3.8 yet. You can still install them but you need to have compiler and their dependencies installed.

The length of release cycle is irrelevant for this particular problem.

> The length of release cycle is irrelevant for this particular problem.

No because you need to compile those packages more often, which as it seem, very problematic.

Those packages do need re-compiling, but the official recompiles are blocked by breakages in 3.8.

One of the issues blocking scipy release is a bug in 3.8's multiprocessing library: https://bugs.python.org/issue38501

Technically they could probably patch around that in their test-suite... but it really is regression in the language.

I see, I wasn't aware of that, my bad.

Binary wheels aren't out from everyone you mentioned yet (they are for NumPy and SciPy... but the maintainers of most of the packages you mention appear to usually wait for their next releases before getting the wheels out), but a NumPy + SciPy + Matplotlib + Pandas + Scikit-learn setup was functional before 3.8.0 was released (I had one spinning during the betas, one just has to be willing to compile things from source usually).

> Binary wheels aren't out from everyone you mentioned yet (they are for NumPy and SciPy...

https://pypi.org/project/scipy/#files sorry, there is no 3.8 binary wheel for SciPy.

You're right, I mixed up scipy and pandas.

Can someone please shed some light on why 12 month release cycles are necessary for a language?

Why would we want our language to constantly change?

No language is perfect, so languages either improve over time or lose to other languages which do.

On why fixed release cycles are a good thing, I find this FAQ by Herb Sutter of C++ fame enlightening: https://herbsutter.com/2019/07/13/draft-faq-why-does-the-c-s...

It doesn't make them necessary, just likely better than the alternatives (ie. no change or changes “when it's done”).

Given that the pace of technological innovation cannot be stopped and all existing technologies have flaws, the alternatives are:

A) Accept that your language will change over time and manage it.

B) Over time, keep changing to better languages after your language dies out.

Yeah, I think A would be the preferred option.

Because objectively our languages are still quite awful and we still have a ton of progress to make.

Look at Python or JavaScript or C++ 10 years ago. They each encompassed the same philosophy they encompass today but lacked a lot of the ergnomics and tooling that make them better languages.

Look at Go or Rust 5 years ago. Look at Java. We are constantly improving our languages because improving a language is a great productivity win.

Something can't be objectively awful.

Awful is a subjective descriptor.

There are several objective measures for programming language "awfulness". The ones I've seen published the most in papers are:

- Time to identify bugs in code

- Defects in written software in fixed exercises

- Defects in written software in real code

- Time to develop a certain program

I can't say that the measurements of these is easy - but when we measure these with old vs. new languages we see change.

One can claim something can't be "objectively good or bad" in the Popper sense of science only caring about falsifiability or in the Quine sense of "just predicting" but that becomes a philosophical argument and debate.

In my opinion since languages are how we express ourselves and we write code in a _very_ roundabout way in many cases - languages are in fact objectively awful, the number of defects is huge, development time is long and developer experience is something few people deal with.

The literature on developer experience and programming language API is just starting to emerge and we are still very very early on.

And of course all the research on this is pretty bad, on grad students / biased samples and the methodology isn't great but it's better than nothing and we should strive to improve it.

If anything that just shows how bad things are. (Although I believe they really are improving)

> One can claim something can't be "objectively good or bad" in the Popper sense of science only caring about falsifiability or in the Quine sense of "just predicting" but that becomes a philosophical argument and debate.

Yes, one can claim such a thing. However you don't have to debate it you have the foresight to not make claims about the subjective qualities of the objective world.

Counter-example: https://en.m.wikipedia.org/wiki/Whitespace_(programming_lang...

Thank you! That page links to a funny Bjarne Stroustrup (April Fools) paper I hadn't seen on overloading whitespace, Generalizing Overloading for C++2000.

http://www.stroustrup.com/whitespace98.pdf

Even in taste, some things are objectively awful -- eating shit for example.

Whether some bizarro outlier likes it doesn't matter, the statistically significant majority doesn't. So, it's not that shit is subjectively good or bad taste it's that said persons are objectively weirdos.

In other fields things are subjective, but people who know better have a canon, which is a more objective ranking. E.g. how Vanilla Ice is worse than Bach in substantial ways, even though some (even many) might prefer the former.

Programming languages are also part of computer science, which has some downright objective criteria too.

Your comment inspired an interesting thread about how nothing can be objectively good or bad. ..until it seems the only thing that is or can be objectively bad is the GP's use of language.

Well, ok, maybe his language wasn't objectively awful, but just philosophically confused, not efficient or sensible use of words for his intended purpose. If you elaborate on that sentence, you will both be able to agree on what was actually meant, coming at it from each side.

I don't think language can be objectively bad. It can be more or less effective in communicating an idea.

I just wanted to help him see the world in a way that is more similar to the way I see it. I think the way I see the world is a good way to see it.

I spreed my perspective like a virus.

I think Python 3 is a less cohesive and worse-designed language than Python 2 in many ways. JavaScript and C++ should have been abandoned.

And yes, let’s look at Java. Scala, Clojure, and Kotlin were all exciting new languages at one point that maintain backwards compatibility by sharing the JVM, obviating the need to make Java itself better—though I will admit that Java itself has improved a hell of a lot.

Java itself has improved a lot taking inspiration from Scala and other languages itself and adding a lot of features (streams lambdas etc).

Worth mentioning Kotlin, Scala and Clojure also improved and changed _a lot_ in recent years and so did Go and Rust and other "new" languages that aren't 30 :]

Nothing would make me happier than to see JavaScript and C++ abandoned (two of the most popular programming languages and the two I use the most except Python probably). However that would require a lot of work to accomplish and such huge undertaking is usually done a lot more incrementally. Doing it in a "big bang" fashion would be quite a lot of work.

That is, JavaScript and C++ are very different than they were 10 or 20 years ago - not just in terms of tooling but also in terms of the language itself.

I understand improvement, all right. I am not against that. What I feel is wrong here is that we get new features every year. A language should be a minimum set of features that everyone uses and not 'all features everyone would ever need'

and long term stability and backward compatibility is a must for a language which is as famous as Python

taking something awful and making it less awful is different than making something good better.

python is 2 is a good programming language by any standard , i would like to see a fork of python 2 with emphasis on LTS and performance optimization.

The whole reason why python became popular is because it had a very shallow learning curve. introducing new cryptic language features every year Is not going to make it easy for new comers. I can say with some certainty that no one is going to use a pure python implementation in production(atleast not yet).

pypy is going to continue supporting python 2 indefinitely. They focus on that LTS release and care about performance.

its been a long time since i used it . is it stable ? it anyone using it in prod ? does it have a decent packaging solution ?

Yes, it's stable.

Yes, people are using it in production. (Examples at https://www.reddit.com/r/Python/comments/bv50uz/is_anyone_us... ).

Is 'pip' a decent enough packaging solution?

cool i will check it out then .

I absolutely agree with this sentiment and seeing Python move further and further from "There should be one obvious way to do it" scares me.

I totally get why you'd want a language with async iterators and typings and why you'd want a language with easy data science interop and easy CPU - I am just not sure they should be the same language and I think it should be possible to learn how to do one (e.g. data science) without the other (high throughput async programming, typings etc).

I really hope the smart people behind the helm have the foresight to discuss the implications of language size. As someone involved in the process in JavaScript both as Node core and as an individual this deeply concerns me there as well.

> We are constantly improving our languages because improving a language is a great productivity win.

I think that ignores the time invested in the upgrade churn and making sure all the little pieces stay compatible.

Home OS releases have moved to yearly release cycles and it drives me nuts. AirPods Pro, released in the last few days, requires macOS Catalina 10.15.1 for full compatibility. A lot of software hasn't been updated (and likely will never be updated). I don't mean to single out this one thing, there are countless other examples.

Businesses I've worked at have kind of ignored these cycles. CentOS seems to have 1-2 dot releases a year and most places I've been jump to the new one every 3 years or so (major releases even longer).

I don't have recent experience with Python because my industry is still stuck on 2.x. The transitions between 2.3, 4, 5, 6, 7 were fairly smooth, but at that pace of new versions and code changes it was pretty common to straddle 3 versions and stumble on bugs fixed long ago in major versions we weren't able to adopt yet.

It sometimes feels like we can never make progress in higher level tools because we're trying to keep up with lower level tools.

> AirPods Pro, released in the last few days, requires macOS Catalina 10.15.1 for full compatibility.

With Apple, I suspect this is intentional to force you to upgrade. It may not be an active decision, but there is probably zero internal pushback when it happens.

Python designers should learn a thing or two about how C++ and Java people manage their language updates. It makes no sense to use a language that stops supporting features like Python wants to do.

Python is not just the language, there is a standard library that is part of the distribution. That library requires bug fixes and feature updates.

Yes, but what good is a language which was easy to use and one which is now introducing a new feature every year? Do you think everyone will upgrade? At work, we are still stuck with Python 2.7 with no intention to upgrade despite what.

English constantly changes. We invent new words to encapsulate new concepts that we didn't have before. It's kind of necessary because the world keeps changing. Why not computer languages too?

Hi, Łukasz Langa here, author of PEP 602 which documents the change. I can answer questions if you have any.

I like black. Thanks for black.

Me too. Black is fantastic.

Full thread from the mailing list:

https://www.prettyfwd.com/t/ERcYpP8xQt-LG82eEoWB1Q/

Is there actual benefit to maintaining a language that isn’t backwards compatible? I remember python being lauded as the best language ever 10 years ago. Now, as I have to maintain several versions or face massive efforts to port code, I dread working with it as much as national instruments. It’s not just a little fractured. It’s fractured to its core to the point that just having any system working feels like a miracle. Why are they steering the ship into every possible rock?

Are you referring to the Python 2 -> 3 fiasco? If so, I think that the Python language committee will never make such a mistake again. They've seen first-hand how difficult it is to get the ecosystem to move forward after a breaking change like that.

But within Python 3, I think the deprecation system is decent. As far as I know, it really only occurs with "uncommon" APIs, so the vast majority of code will continue to work as expected on later Python 3 versions.

> But within Python 3, I think the deprecation system is decent. As far as I know, it really only occurs with "uncommon" APIs, so the vast majority of code will continue to work as expected on later Python 3 versions.

The problem is that every new Python version seems to break something. Python 3.8 was released almost two weeks ago, and some popular libraries still don't work with it, e. g. SciPy, Matplotlib, OpenCV, PyQt, PySide2 and probably a bunch of other libraries that depend on the above.

Which is just about everything outside of Keras/Tensorflow that I use. Highly frustrating. Languages should be designed with backwards compatibility in mind right from day #1. Porting code isn't free.

Mostly, yes, as I’m responsible for a data acquisition system written 10 years ago. “Just” getting to python 3 is a massive effort which would first require getting to 2.7 from 2.5, which itself would be a time consuming task. I’m starting to think the best solution would be to just stay on python 2.5. Updates to software should not be the implementer’s tormentor.

Then this is an unfair critique of this new release process. If you had issues with breaking changes between minor 3.x versions, then I'd fully understand.

Are complaints about things breaking between 2.x versions somehow invalid? It’s the same steering group that has made a series of breaking changes. Even people on branches of this thread claim there are breaking changes in 3.7 > 3.8. I’m not sure I would willingly use python for any new project unless necessary, putting it into the same bin as national instruments.

It's a valid complaint on it's own, sure. But it kind of looks like you're complaining that they're going from an ~18 month cycle to a scheduled 12 month cycle. It's not really valid as a complain for this particular change. You're talking about a decade. 18 months vs 12 months seems inconsequential for you.

I'm complaining about the ethos of the python maintainers. I don't see how putting out updates more rapidly will do anything to my complaints other than make future me dislike using python even more. At first I thought they missed a beat but now I think they're playing free jazz.

The 3.x releases _are nothing like_ the 3.0 release. You have a valid complaint about the 3.0 release, which was 11 years ago. If you've not tried updating from 3.x to 3.y — which is a very different process — then you have no stake in this change.

It's your call to make. If your system doesn't have some sort of 'security' risk to it, then all the reasons you have to upgrade are purely internal cost-benefit analysis.

Code is a tool, treat it like any other. Just be aware of possible accumulating deferred costs. I don't know exactly what field your in, but just imagine any other piece of equipment or component going EOL, and the manufacturer only providing semi-drop in replacements. Or even if they are advertised as "drop in", you know that half the time they aren't, and something derpy happens on integration.

The system is not firewalled from the internet. There are tools that need internet based licenses. There are organization firewalls, but there is always a risk involved with having unpatched platforms. I have to weigh that risk against the cost of porting. Both are difficult to gauge.

How are the Python 2.5 components being invoked? Are they python processes that are constantly running? Or are they jobs spun up by other processes?

Oh they’re always running and on dozens of machines.

If you still haven't upgraded from a version released 13 years ago to one released 9 years ago, what difference does a slightly faster release tempo make? If it's that much work, something else has gone wrong in the software development process and all the Python upgrades are doing is making that failure harder to ignore.

I think there's also an effect where some of us have gotten used to Python 2.7.x being utterly predictable and unchanging for like, the decade since it came out.

It's not Python's fault, but for example I was trying to get something running on Ubuntu 16.04 (Python 3.5) where the primary developer of it is running Ubuntu 18.04 (Python 3.6), and it seems like 3.5's version of the asyncio module has a bunch of bugs that make the tool in question basically unusable.

I know the usual advice— install newer Python from deadsnakes, use a virtualenv, etc. None of this stuff is the end of the world, but it can be jarring if you're only now getting off the Python 2.7 train and used to everything just working everywhere.

That's not really fair, though. If the developer were writing it on 2.7 and you were running it on 2.6, I wouldn't hold that against Python 2.

Oh I completely acknowledge that. This is less about Python and more "fast moving platform" vs "slow moving platform".

well asyncio was introduced in 3.5 though, it just happened that initial version was buggy and you should assume asyncio is available since 3.6 (or actually 3.5.3+)

I don't think it was a mistake. There were some intractably difficult, "rip-the-band-aid-off" types of changes that had to happen at some point. The PSF decided that 3.0 was the right time to make those one-time changes so that future migrations could be as small as possible.

Then why didn't they remove GIL? Given the end of Moore's law and Python's performance, this alone would have justified the discontinuity.

Because GIL is useful, preventing you from shooting yourself terribly in the foot among other benefits. Languages with better concurrency support (main reason for removing GIL) were designed from scratch for that purpose. You can't remove GIL and keep Python being Python everyone knows. And if you don't mind using another language, there are many choices.

I believe Jython offers the same level of protection against shooting yourself in the foot as CPython, yet it allows allows for parallel execution by relying on finer grained locking. The concurrency issues with threads are still present in CPython with the GIL as the interpreter may switch threads between any instruction.

There've been various attempts at this over the years, but it's really orthogonal to changes to the language itself.

Also, check out the SharedMemory objects[1] in 3.8. For my own use cases, this goes a long way toward making me no longer care about the GIL. I'd rather work on multiprocessing code than multithreaded code any time, and if you can easily share objects across processes, then the GIL isn't relevant to anything I'm working on.

[1] https://docs.python.org/3/library/multiprocessing.shared_mem...

I would argue the GIL is a feature put in place to make it easier to write parallel code.

If you want to do this in a more performant way you should be looking at a different language.

People talk about the GIL as if it’s just some drawback that could be removed and everything would be faster. There is a very good reason why it’s there.

> I would argue the GIL is a feature put in place to make it easier to write parallel code.

It doesn't do that though. The GIL protects interpreter internals. It doesn't make Python code thread-safe, although it does make various operations atomic:

* native calls (which doesn't release the GIL) * individual bytecode instructions (which don't call into more python code)

The latter can make it seem like it magically protects your code, but even a simple method call is at least 3 instructions (LOAD_FAST, LOAD_METHOD and CALL_METHOD). Each instruction is atomic and the method call will be if it's a native call (e.g. dict.get) but that's it, the entire call itself is not atomic and you could thread-switch between LOAD_FAST and LOAD_METHOD, or LOAD_METHOD and CALL_METHOD.

You just explained a bunch of implementation details but at the end of the day my point remains: having the GIL makes it easier to write parallel code. I didn't say it magically makes everything threadsafe.

> having the GIL makes it easier to write parallel code.

It does not. What it does is create a bigger trap, because the code looks to be working in more testing situations, and will be more difficult to debug when it starts failing.

Isnt the GIL part of the implementation rather than the language?

In the same exact way the garbage collector is. You never touch it but the fact it's there makes it easier to write code.

Assuming you mean concurrent rather than parallel as the GIL prevents you from running Python code in parallel. How does the GIL make things easier compared to the finer grained locking found in Jython which allows parallel execution of Python code in threads?

Threads may still be switched between instructions every sys.setswitchinterval milliseconds so every instruction boundary is still a potential thread switch point, leading to all the issues described by Glyph in his Unyielding essay: https://glyph.twistedmatrix.com/2014/02/unyielding.html

Because the GIL is only a problem in specific sets of problems that the majority of python users won't run into. With the addition of asyncio etc, the negative of GIL is usually pretty restricted to CPU-heavy, blocking services which the core team decided was a very small subset of users. So removing the GIL would require an extremely heavy rewrite of the way python works at a fundamental basis, and it would break a lot of the "promises" that python affords in regards to memory management and safety.

As an aside, there is currently a PEP that will allow for better concurrency __without__ removing the GIL, [PEP554](https://www.python.org/dev/peps/pep-0554/)

and a larger project overall for multi-core support, which is (hopefully) being included in 3.9 [Multi-core](https://github.com/ericsnowcurrently/multi-core-python)

Removing GIL will break ecosystem in worse way than Python 2->3 ever did, as both interpreter and compiled libraries rely on it for thread safety.

There are other paths for improving performance that Python is embrancing: asyncio provides considerable performance gains to IO bound apps, and memory-sharing between processes allows for relatively cheap sidestep for those who find threads too slow for their use-case.

In addition to the points other people raised, "simply" removing the GIL by changing the current reference-counting GC to do atomic reference updates will also substantially reduce single threaded performance; there was a PyCon talk about that.

https://www.youtube.com/watch?v=P3AyI_u66Bw

See also Swift for another language whose implementation has performance issues caused by ubiquitous atomic reference counting.

Aside from Unicode strings, what changes actually required breaking backwards compatibility.

For that matter, why did the switch to unicode strings require breaking backwards compatibility? I don't understand why we couldn't just move to assuming all strings are utf8 unless specified otherwise. Keep [] byte oriented, add a .at() method to index by code point.

The reason for it were bad decisions made in older version of python. The problems were:

- conflating text with bytes, python had no way to tell whether given string is a text or bytes, because in 2.7 was the same thing.

- introducing unicode as an unicode type, this essentially made the problem worse because in addition to mixing text with strings, they added extra type to represent text, which was optional and some people used unicode some didn't.

- a cherry on top was implicit conversion, so if you passed unicode type where str was expected python implicitly converted it, and vice versa

This basically resulted in people writing a broken code in python 2, code that worked fine with us-ascii, but randomly blew up if there ware some non standard characters processed.

In python 3 instead applying another fix, they decided to do it correctly from the start. So str (Guido actually regretted he didn't called it text) is representing text and bytes are representing, well bytes. Python 3 also does not do implicit conversion as well.

I actually think they could perhaps help with migration, by adding to 2.7 one more import to __futures__ that disables implicit conversion and then treat bytes type as a distinctly different type than str. That could help people fix their code in 2.7 before migrating it, but anyway it's already too late to do it, also there is a hack that you could disable implicit conversion in python 2, but it showed that stdlib also relied on it heavily, so fixing that maybe was not worth it.

> There were some intractably difficult, "rip-the-band-aid-off" types of changes that had to happen at some point.

I suspect you're referring to Unicode here. In that case, I think they could've just added a flag to Python 2's str type to indicate "is UTF-8" and deprecated the old unicode object. Then add some functions to extract code points or grapheme clusters or whatever else you need from the old school str object.

I might be in the minority, but I really like that Python 2's str could hold arbitrary binary data, of which UTF-8 is just one possibility. It had good interop with C, which I think is fundamental to a glue language like Python. I'd rather have fewer string types instead of more (one of my complaints about Rust too).

If you meant the print function, there were other ways to solve that too. The simplest might be to create a new name for the function and deprecate the print statement. So old code uses the statement "print 123" while new code is encouraged to call the function "echo(123)" or "ouput(123)". Bikeshed the actual name...

Note when I say "deprecate", I mean provide a timeline over several releases where it continues to work. Then issue deprecation warnings which can be silenced.

All of the newer features in Python 3 (@ operator, async/await, type annotations, etc...) could've been added in a mostly backwards compatible way. (Note: adding async wasn't really backwards compatible even in Python 3).

Anyways, hindsight is 20/20, but I really do think the path for Python 3 was a poor choice in comparison to other options.

Your proposal only works well for US ASCII users. What if I want to manage multiple ISO-8859 encodings in conjunction with 7-bit ASCII? Maybe I also have some EUC-JP multi-byte text to deal with. It becomes an intractable mess without explicit encoding management. Someone will absolutely end up misinterpreting encoded text as bytes and cause all manner of compatibility and security issues. Having a Unicode string type forces this to be dealt with even if it is inconvenient when taking in data from outside the Python environment.

> Your proposal only works well for US ASCII users.

No, and I explicitly mentioned UTF-8. My suggestion is that str holds arbitrary immutable binary data and that you have a method which can interrogate whether that binary data is valid UTF-8.

Yes, real world text is messy and there are lots of encodings, compression schemes, and exceptions (UTF-8 with byte order marks, overlong encodings, or surrogate pairs, as examples). If your main task is converting text between outdated or broken encodings, I don't have any problem saying you need a separate library and shouldn't burden the rest of the user base. Despite it's flaws, the majority of the world has settled on Unicode with a UTF-8 encoding.

"Special cases aren't special enough to break the rules."

> I really like that Python 2's str could hold arbitrary binary data

Python 3's str can also hold arbitrary binary data. That ability was introduced in PEP 383.

Thank you for telling me about this - I didn't know they did that...

My first thought as I started reading the PEP was, "Why did they bother adding the 'bytes' type if 'str' is just going to be able to hold everything anyways?"

After looking at more of it though, it seems like they're storing the binary octets as code points in one of several internal Unicode representations. Moreover, they're abusing (reusing?) the range of code points reserved for 16 bit surrogate pairs, but only using the low half of the pair. This is all clever in the bad way.

This seems like a real lack of taste to me, and I doubt the Guido from 1991 would've found it acceptable to have 'str', 'bytes', and 'bytearray' the way they are. (Let's ignore 'buffer' became 'memoryview' for now...) It used to be a simple and elegant language.

How was it a fiasco? The old version was supported for free for what, a decade? Usage of python is skyrocketing. I don’t know anyone who honestly prefer 2 over 3. The number of python packages released in one year now is probably higher than that of all years combined up to the release of python 3.

The process was not perfect, but in my book it’s a success.

It was a fiasco because it wasn't possible (until much later) to support both Python 2 and Python 3 in the same code base. This made for a gridlock between dependencies and users.

This is a misconception that has likely been compounded by the initial advice (by core Python folks, who eventually reverted their stance) on not doing this. For example, see this blog post https://aroberge.blogspot.com/2009/08/crunchy-10-released.ht... from 2009 for a non-trivial program that supported Python 2.4, 2.5, 2.6 and 3.1. It was a bit of a pain as it dealt with lots of unicode/strings, but it was doable. It became easier to support both Python 2 and Python 3 later on, especially once Python 3.2 was released.

Ok, sorry, replace "not possible" with "very difficult and not recommended."

I don't see how that changes what I wrote in a very meaningful way.

It's much better for things to be released when they are ready than on an artificial and completely arbitrary cycle, brutally enforced.

i think type obsession should stop . And more effort should be placed on performance optimization of exiting apis. but there is lot to be learned from LTS lifestyle of other products like debian and redhat which allow community to be built around them.

> I think type obsession should stop .

Typelessness is the bane of Python. The community would be wise to finally take it seriously instead of keep pretending that type checking wouldn't fix whole classes of the bugs and lead to safer software, higher productivity, and overal better dev experience.

Yeah, I consider myself an advanced & "bought in"python programmer, but recently I've been dabbling in VSCode + Typescript and it is truly amazing how much more confident I am in the things I create, when I'm careful to use the type system and things like Union Types to my advantage.

Python typing still lags behind, and my python code is the scarier for it.

Also -- one gamechanger for me has been the addition of an IDE as the go-to environment for programming in strongly typed languages... having an IDE that "gets" the language you're coding in, and its types, has allowed me to eschew a huge number of my "todo" statements altogether -- as I'll add a signature / typename where I'm going to use it, and the compiler won't let you do anything until you've actually done what you asserted you were going to do.

I use python's type annotations, but didn't use TypeScript, what kind of things you mean when saying that Python lags behind?

Python doesn't have a transpilation stage like TypeScript, which gives JS a sort of compile-time check that forces the developer to fix type errors. Python type hints can just be ignored.

That's a really silly argument.

"TypeScript is better because I'm required to run the transplier"

If you believe that type checking is necessary, you won't have problem running mypy.

Also TypeScript has a transplier, because there was no other option, browsers use JavaScript and don't understand TypeScript so the conversion is necessary. Maybe that won't be needed in the future, but it is needed now.

A lot of people don't only work on their own code. They have dependencies, they have coworkers, they have to work on code they wrote nine months ago. So it's not "I need to be forced to use the type checker," it's "everybody needs to be forced to use the type checker."

If it's required, you add mypy to your CI. Have it run whenever a PR is submitted.

I have no idea what argument/point you're trying to make.

The TypeScript workflows happens to include this amazing process where auto-complete and other features work on non-TS libraries/code for no other reason than someone decided to take the time to provide type information on something that otherwise wouldn't have it. In a repository/workflow that is easy for any developer to easily tap into.

This is amazing.

How did you jump to "TS is better" or some silly thing like that?

TS build pipelines almost by default don't let developers build/commit code if a type definition isn't followed. This is a HUGE workflow step that isn't common with Python and type hinting. It makes a big difference. There is no Python equivalent of MyPy "headers" or type information for countless third-party libraries.

I use flake8/Black as part of my Python development experience, but it still doesn't compare to what the TS community has been able to pull off.

> The TypeScript workflows happens to include this amazing process where auto-complete and other features work on non-TS libraries/code for no other reason than someone decided to take the time to provide type information on something that otherwise wouldn't have it. In a repository/workflow that is easy for any developer to easily tap into.

> This is amazing.

The same thing I observe in Python and PyCharm.

> TS build pipelines almost by default don't let developers build/commit code if a type definition isn't followed. This is a HUGE workflow step that isn't common with Python and type hinting. It makes a big difference. There is no Python equivalent of MyPy "headers" or type information for countless third-party libraries.

I don't think I understand, could you give me an example. I don't see how types can work if no type information is provided.

PyCharm and JS (Babel) are only similar in that they turn text source code into AST's

With Python + PyCharm, the default mode is to allow developers to exclude and ignore types and code will still run, no matter how many orange/red markers are in PyCharms gutter. To change that mode, you need to consciously decide to add something to your workflow that will impede you if something is wrong e.g. git hooks to run linters, flake8, black etc.

With TypeScript, the default mode is to stop the developer in their tracks - the TS compiler will fail, and you can't run code (generated JS) that doesn't exist. Types are not optional, unless you make them explicitly so (which is still providing a type).

> I don't see how types can work if no type information is provided.

TS can still import native JS code... given that it has a TypeScript declaration file [0]. Many popular libraries offer one now, but many more do not. The community came together and created DefinitelyTyped [1], where 10k contributors have made 65k commits to provide TS declaration files for over 5,200 libraries spanning countless versions/releases.

This means a TS developer can import JS code that doesn't have 1st-party TS support, but does have 3rd-party through DefinitelyTyped.

[0] https://www.typescriptlang.org/docs/handbook/declaration-fil...

[1] https://github.com/DefinitelyTyped/DefinitelyTyped

> "TypeScript is better because I'm required to run the transplier"

You've missed the point entirely. The whole point is that TypeScript benefits from compile-time type checking, which is performed by TypeScript's transpiler.

Python's type annotations are just that: annotations, which can/have no effect.

When you have a compiler/transpiler the process typically is done in two steps, running a type checker on the code and then the actual compilation (which could be split into more steps). When you run mypy, you're doing the first step.

The types you define in TypeScript you similarly define in Pythin, they just call them annotations, and there's an option to reference them at runtime, but mypy doesn't work at runtime (perhaps that's where the confusion is from?)

my whole gripe with a strict type system , is that it does not need to be a language feature the same way tests/IDE dont need to part of the language.

you can add them if you need it but it shouldnt be a center piece , what should always come first is the feature / tech spec.

Python has a great type system aswell its objects all the way down. And you can mix and match how you see fit. If you have identified that type checking is crucial part of the tech spec as a sanity check or otherwise you can import type checkers(ie mypy) with a very focused api to solve that problem.

And in my opinion type obsession obscures the techspec rather than clarifying it , and places a large burden when the question of changing the spec comes. The simple cognitive load of keeping track of all the int's which need to be made to str's can drain you and make you loose site of the functionality.

Why's that? I love the type system. We're not dogmatic about it - like, no one's going to reject a PR because it doesn't have complete type coverage - but it's a seriously useful tool and has uncovered some long-hidden bugs in our code.

From a tooling point of view, I love hovering over a function name and seeing the types of its parameters. PyCharm supported that, using what I consider to be ugly and janky docstring comments, but now every editing environment can provide that information.

Yeah, it also helps with auto-completion and makes refactoring reliable.

From my experience type annotations in python are actually more reliable than types in Go.

Can you expand on that? I'm not familiar with type annotations, but I use Go daily. Aside from interface{} abuses, how can something be more reliable than accepting a type in Go? (int64, string, []byte, myType{})

I basically had less issues when trying to refactor annotated python code in PyCharm than Go code in GoLand (if you're not familiar, both of these IDEs are made by the same company JetBrains).

My belief is that it is a bit harder for IDE to determine which structs and interfaces should be changed, due to Go using "duck typing" (or more correctly called structural type system).

The mypy uses nominal type system (similar to what you're used in C++, Java etc) by default but it also allows to implement duck typing in places where you need it through Protocols.

Overall the type system is more powerful than the one in Go[1], it even includes generics =)

Slightly unrelated, but when working with Go, I feel like the type always gets in my way. Go typing (similarly to Python) is strict and that's great, but Go trying to be low level it implements different sizes (int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, float32, float64) it doesn't do implicit conversion like C, but it also doesn't implement generics, and stdlib only implements functions for one of the type. That means if you use anything else than int, int64, float64 in your program you'll be in a lot of pain and will be constantly casting the types back and forth.

Having programmed in Python (with type annotations) and now at my current job in Go, I love Python.

[1] https://docs.python.org/3/library/typing.html

Typescript has transformed the JS community, even for those who aren’t in the TS community at all as most major libraries have community updated typing which your IDE can consume.

You don’t have to do any typing to consume some of its benefits; someone else can pay that cost for you. You don’t even have to be aware that TS exists.

> You don’t even have to be aware that TS exists.

I never thought of it this way, a very interesting insight.

TS has introduced an entirely new type (pun intended) of FOSS that isn't even code, it's just header files to let you know what the inputs/outputs of things are.

An observation that is nearly impossible to make is that JS is something like 20 years behind the rest of the world in so many ways. Now that it has moved on from being a little glue scripting language and people are building complex software with it, all the same problems that Java or C++ or Unix have encountered in the past are coming home to roost, and more structure and tooling is required.

I like Python because it is dynamic but recently I have been catching at least one bug per PR whenever I changed an existing python code.

I too was sceptical before using it and Python or the community does not enforce it. But it is indeed useful.

Welcome to the world of Ruby. It has a new major release every Christmas. On the day.

Whatever you write in python,you must commit to maintaining it yearly. If you write software and people buy it,you have to update it once in a while.

There are many use cases where this won't fly and it's fine to use something more standardized and stabilized. They should have done this a long time ago to avoid the 2.x -> 3.x headache.

Undesirable for those of us that sorta depend on stackoverflow answers to get a lot of things done but I am sure there's a bigger picture the python devs are seeing.

Reply on news.ycombinator.com