Python 3: Ten years later
archive.fosdem.org
archive.fosdem.org
On the other hand the core development team "outsourced" many of the issues that people actually want to have solved to the community like packaging and distribution.
Python's biggest downfall is a highly academic approach with slow release cycles which makes large feature development hard and detached from the real world. Python 3's unicode model is completely outdated but was supposed to be the big feature of the version. While that was happening however the rest of the world started moving over to UTF-8. There was no feedback loop that could have enabled Python to be UTF-8 based. It was released and then decisions were locked in.
Similar things happened in asyncio. New versions of it only land with every major version of the language which meant that a lot of asyncio only became really useful with 3.6 and 3.7 which was years after it was initially created.
I have very high hopes that the new leadership of the language will be able to tackle some of these problems. Python 3 is way less backwards incompatible to 2.7 now than it was 10 years ago. The two versions moved closer to each other and today it's pretty easy to support both from a single codebase. With faster iteration we might have been able to cut this time short by 5 years or more
I’m not sure what you mean. Are you suggesting that Python use UTF-8 encoded strings internally instead of decoded Unicode codepoints like it does currently?
I’ve worked on Perl that uses that approach and it’s an absolute nightmare. All string operations suddenly must be aware of details of UTF encoding, which complicates even simple things immensely. You get hard-to-find bugs when some code somewhere introduces invalid UTF to a string. The only real benefit you get is slightly smaller memory footprint when dealing mostly with English text.
I much prefer Python’s decode-process-encode approach.
perl's problem is that they didn't make room for the original encoding, it's only latin-1 or utf-8 from everything else. so the context needs to store it, like IO, process. and for performance they silently loose the utf-8 flag for 128—255 codepoints.
but all this is still better than the libc/POSIX design failure to use ENV vars, where the encoding is global, not localized. every single string API needs to do a getenv, and its interpretation is action at a distance.
There’s no other way, is it? You basically have to have the complete string library re-implemented for each supported multi-byte encoding.
The only trouble is that you have to remember the original encoding, if you need that information. It's stored nowhere. E.g. to write to a file in the original encoding you have to open it with the correct IO layer. perl6 got that fixed.
Correct. Because the current approach completely fell apart when people started using emojis. The internal encoding in too many cases is now always UCS4 which is very wasteful.
> All string operations suddenly must be aware of details of UTF encoding, which complicates even simple things immensely.
Go, Rust and Swift all show that using UTF-8 internally is not hard. Not sure why you think its hard. I have written a lot of code in those languages now and I feel much happier with their unicode support than Python's.
It's not dead, but as of now, it's mostly relegated to legacy code and a dwindling number of holdout programmers who refuse to adapt based on some misguided principle. It was a long and painful process, but existing libraries are dropping support for Python 2, and new libraries are created daily that will never support Python 2.
That said, I still regularly interact with people who are new to Python but have been indoctrinated somewhere with the belief that they must use Python 2.7. They've heard it's more stable, faster, or Google uses it. We discuss some of the actual tradeoffs, and they're quickly disabused of the notion that Python 2.7 is a good option for new Python projects today.
But I totally agree, some Python 3 additions (I'm looking at you, asyncio) have been handled in an unfortunate way.
Makes me wonder how much a problem this is and where.
Already we have much of the community still sitting on Java 8 even as the new long term release JDK11 has been made. It wouldn't surprise me at all if JDK8 becomes the Python2 of the java world: 10 years from now, there will still be whole segments of the community using it.
People constantly complain, but honestly changes like Python 3 have to happen - the alternative is just another completely new language taking over instead.
Write a Python 3 program that takes two arguments, reads a text file named "input.txt", replaces all instances of the first argument in the text with the second argument, and writes the result to "output.txt". For extra credit, add an option to print to stdout.
My bet is very few people would write that program correctly.
This is the most straightforward way I can think of to implement the program you described. I think this is more or less what someone would come up with after glancing at the Python tutorial:
https://gist.github.com/avian2/e3968bf933212ee561410b15495e6...
What is wrong with it? It works as expected with non-ASCII characters in both command-line arguments and in input.txt (Python 3.5.3 on Linux, LANG=en_US.UTF-8). Yes, it doesn't work if you have a file that's not consistent with your locale, but you can't really blame Python for that.
C:\Users\local\tmp>python --version
Python 3.7.1
C:\Users\local\tmp>python foo.py Diet Parliament
Traceback (most recent call last):
File "foo.py", line 7, in <module>
text = f.read()
File "C:\Users\local\AppData\Local\Programs\Python\Python37-32\lib\encodings\
cp1252.py", line 23, in decode
return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 107: char
acter maps to <undefined>
Imagine I wasn't a developer who has spent dozens of hours on fixing encoding problems in Python. Imagine I'm just some person using some software I've been given and this is what I encounter. Can I actually complete my work? Is this a good user experience?Previously, if you were a UNIX-y scripter writing UNIX-y scripts on your UNIX-y OS, the fact that Python just kind of pretended encoding issues would never exist was a help to you. It adopted the same "everything is ASCII, or at most UTF-8 in the ASCII range, and if it isn't I'll break in cryptic ways" approach as most other UNIX-y scripting things.
If you were doing anything other than UNIX-y scripting on your UNIX-y OS, this easily became a huge nightmare in Python 2. Django went through a massive rewrite early in its history precisely because of this, to ensure that encoding/decoding happened at the boundaries and everything you'd work with inside a Django app was already a Unicode string. And I remember what it was like trying to work on the web before that approach, and what the work to fix it was like.
Python 3 decided to make the UNIX-y scripters actually learn what a horrid mess UNIX is with respect to locales and encodings and filesystem paths, in order to free the rest of the Python community from the nightmares inflicted by prioritizing the UNIX-y scripters to the exclusion of everyone else. So yes, you have to do more work. Yes, you have to learn that a file path is actually an opaque bag of bytes that may not be in any actual encoding and thus can never properly decode to a string. Yes, you have to learn to use fsencode() and the surrogateescape handler in order not to blow up your scripts.
But I'm OK with that, because it puts the workload on you when you're using such a system, rather than magically trying to fix it for you at the cost of everyone else's sanity. It also means that you have to learn to write those scripts correctly. Which is more work than what Python 2 required, but not the world-ending apocalyptic horror it's usually presented as (and is, again, mostly the fault of UNIX-y systems doing their old UNIX-y things, not the fault of Python).
> Python 3 decided to make the UNIX-y scripters actually learn what a horrid mess UNIX is with respect to locales and encodings and filesystem paths
Show me how much Python 3 improves this. To expand on the program before, make it a directory named 'input' full of files, and a directory named 'output' to put the processed files in. Print each file name as the corresponding file is processed to indicate progress.
I would be applauding Python if it did make the difficulties with this exercise obvious, but it absolutely does not. The file system APIs return strings, but the strings they return may not be valid Unicode. PEP 383 turned the Python 3 str type into a bag of bytes.
Python tries to sweep encodings under the rug. It makes the encoding a default value all over the place and hides conversions everywhere.
I 100% agree that developers need to think about encodings and handle them in their programs. That's exactly why I hate string handling in Python 3: because rather than making you handle the corner cases, it pretends they don't exist, until they're found one by one by your users.
Python 3 encourages developers to write broken string handling code.
You just want to fight someone because you're angry, and I don't do that. No matter what someone writes you'll find a way to argue into it being wrong and then prance around declaring "victory".
(I'd also bet that you probably couldn't do it if I were the one who got to set the evaluation criteria, and you also couldn't pass other "challenges" like writing proper HTTP handling -- the person who gets to grade the challenge always "wins", which is why you want to be the person who grades the challenge)
PEP 383 turned the Python 3 str type into a bag of bytes
PEP 383 provided a way to read certain things -- primarily filesystem paths which can be basically anything -- using an escape mechanism to replace non-decodable bytes with surrogates when decoding to string, which in turn allow losslessly transforming back to the original bag of bytes.
Which is necessary, because there are real filesystems out there that really have paths and names that can never validly decode from any known text encoding. It doesn't turn strings into "bags of bytes"; the resulting str still is an iterable of actual valid Unicode code points.
Python tries to sweep encodings under the rug.
As the saying goes, you can't reason someone out of a position they didn't reason their way into, so I won't try here.
Python 3 encourages developers to write broken string handling code.
Python 3 no longer tries to cover for the random gibberish that's legal to put in filesystem paths, and makes the developer handle it. Are there lots of developers out there who don't realize that filesystem paths can legally contain undecodable garbage? Sure. That's not Python's problem to solve, though; it gives you the surrogateescape handler, and the fsencode helper, and keeps working on things like PEP 538 and PEP 540 to try to give you tools to work around it. But Python can't magically fix the mess that is UNIX locales and bag-of-bytes paths (nothing can, short of burning UNIX down and starting over), and doesn't try to do it for you.
Unix has its own problems, but the program avian wrote works correctly on Linux. Most encoding issues I encountered when working with Python 3 were on Windows.
PEP 383 was making the best of a bad situation without breaking the API again. The real mistake was having the functions return strings in 3.0. The operating system APIs should have returned path objects that require an explicit conversion to string with an explicit error handling mechanism.
Python gives you all the tools you need to do this right, but they're easy to unknowingly use in ways that break on corner cases. A well-defined API should guide you towards the correct solution and should make pitfalls obvious.
In any case, I should probably give it a rest. I work hard to make sure my programs do this stuff right, and I suppose that's all I can really do.
Honestly it surprises me how simple and easy it is to understand this trace and I'd agree with the statement that this is a good user experience.
It's better now, less space to screw up.
It’s not that Python 2 couldn’t do it, but the corner cases and error handling was a pain. Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works.
Python 3 was a rocky transision, but I never understood the negativity and I’ll be happy to see Python 2 die, so our system can go back to having just one Python interpretter.
> Trying to guess encoding at runtime was pretty error prone, with Python 3 it just works.
You should not guess on 2.x either. On the other hand in 3.x the interpreter sometimes guesses the encoding for you when working with OS data on unix and that has lead to some absurd bugs when it falls into that surrogate handling fallback code: http://click.palletsprojects.com/en/7.x/python3/#python-3-su...
Python 3’s Unicode handling has been a huge win for cross-platform deployments in my experience. We run a patched version on 3.3 with some backports from 3.4 on millions of machines, and it is actually possible to support non-Latin filesystem paths, where that was just not possible when we were using 2.6.
In terms of an ecosystem, Python 3 forces casual developers to confront Unicode head on, where 2.x the default is to sweep it under the rug.
I’m extremely grateful that a breaking change was made, even if the execution wasn’t perfect.
So for most people it's better. The same goes for the guessing encoding part. Sure, you really shouldn't, but sometimes you had to and then you end up with chardet all over the place. Python 3 may do the exact same behind the scenes. The thing is that the Python core developer are much more talented than I am, so now I get to not think to much about it and just build on their work. From the "regular end-user developer" it's better.
Every single time I need to install something with pip on MacOS there is this hour-long struggle with figuring out how to get the correct version of Python (2.7 vs 3) working with the correct version of pip working with the correct version of the library itself.
I don't know, I guess people who use Python regularly have figured it out, or maybe it's just a MacOS thing. For me, it's just a major PITA even 10 years after release.
It never seemed that hard to keep track of the different versions.
Hell, it even breaks itself:
pip install --upgrade pip # => pip unusable
https://github.com/pypa/pip/issues/5447
https://github.com/pypa/pip/issues/5221
etc
Upgrading pip itself is kind of a special case I think. You don't want to upgrade stuff managed by a system package manager using pip; their work will conflict with each other. This includes pip itself. I think this is kind of a mess and I don't know of a great way to deal with it other than just remembering not to upgrading system packages using pip (yes this includes pip itself, and means you should avoid trying to upgrade via --user if it's installed on the system). On Windows where I am it's ironically less annoying since there's no system-level package manager to worry about... go figure.
On Mac, the system uses python2 so pip and python link to py2.
You can change the system -not fun- or wait for apple to upgrade to py3. Ive been waiting a long time.
There are tools like conda that do manage py2 and py3 projects living in harmony.
What you could do with simple npm command now doesn't work unless you give additional command lines when it comes to cron and other non interactive environments like process monitoring to restart the process.
Or use Anaconda, which is your own Python "Disneyland" but it is hard to interface with external infrastructure
I was not paid for this message :)
Or pip3.6 etc
--user is useful also
Python 3 did enough to break everyone's existing code, but nowhere near enough to make that worthwhile.
It's common for other language communities to consider the Python 2/3 transition as an example of "what not to do".
Somehow the world of JavaScript has been able to quickly adapt to a rapidly changing language and the community is hardly fractured along the lines of language. Adopting new language features and browser APIs is just a matter of fact. What's different about Python?
There's a good reason people wanted to migrate to newer JS and I never heard people say ES6 got something wrong over ES5.
Titles are by far the strongest influence on discussion, so this rule is much more important than it seems.
https://news.ycombinator.com/newsguidelines.html
(Submitted title was "Python 3 Is Now 10 Years Old Was It Worth It?".)
Not to mention "unless it misleading or linkbait", which is not something code can take care of.
Not all title additions are bad.
Edit: not really the focus, it's more of an outline of an overview of what happened. But still.
Makes life a lot easier.
The editor support that I get from typing/mypy is fantastic! It feels like writing in a statically typed language. To me this is by far the most important feature of Python3. It's a godsend for larger/complicated code bases.
Imagine if Ubuntu 14.04 (Trusty) didn't bring Python 2. All devs of relevant stuff would have received a shower of upgrade requests. Maybe 2014 was too early to force this, but I'm sure in 2018 it could have been done with Ubuntu 18.04.
I know this looks like work for the sake of upgrading, without any real benefit for most projects. But I guess it's not wrong either; you just couldn't expect that Qt 3.3 (circa 2004) was distributed 10 years later in Ubuntu 14.04 (Qt is just a library, not a language, but it practically transforms the way you write C++ so it's a valid comparison IMO); however here we are in 2018, 10 years after the release of Python 3, and Python 2 is still the default interpreter in Ubuntu 18.04.
But it survived, which is the important thing. I thought 3 would kill Python, but it has survived and that is a very good thing.
Anecdotally, non-nerds fins python easier to read than other popular languages. It’s become the go-to language for a bunch of domains that reach outside core nerding (data analysis, education, DSL like stuff) and I predict Python will therefore be widely used in 2050, and probably be popular aged 100.
I love python. For me, it’s damn near perfect in many ways. But it’s looking more and more like Perl 6 than what it should have been.
The sooner we get to Python 4 (whatever that ends up being), the better, as we need to move past the v2 or v3 question for developers and users.
Choose the right tool for the job. If you need speed, choose c
No one seems to struggle too much though. And the things they struggle with, they would struggle regardless of the language chosen.
The problem is that this elegant subset of C++ is hidden inside ... the rest of C++.
I guess that's the counter example to Python's 2/3 breaking change. C++ just keeps adding new features while retaining all the old ones for backwards compatibility, even if they're awkward in context of the new language.
Critical FORTRAN libraries are still the center of certain types of analysis, and they're often used by _everyone_ in the field.
The reason for that stems from two issues:
1. Making accurate simulation libraries for things like weather formation, magnetic field formation or hydraulic fracturing requires crazy in depth knowledge of the systems you are modelling, and takes years of implementing and hypothesis testing to ensure they are accurate enough for academic use.
2. Academics are not software developers, they don't get paid for the software they write, they get paid for papers. They get no reward for rewriting those crazy indepth libraries every decade in the new language de-jour.
This is a perfect example of why the python2 to python3 hard-cutoff transition was a terrible idea. "Just update your software" is something you can say to professional software developers with a straight face. You can't say it to academics, or mechatronics engineers, or real estate agents, or photographers, or _anyone else_ who has a job that benefits from a little bit of software but is mostly about delivering something other than software.
Well, you can, but they're going to ignore you.
I wonder, once 2.7 is EOL in a years time, whether the language developers will consider a "3.x LTS" release?