Hacker News Reader

Hacker News

Comment by metalliqaz | Hacker News Reader

Full thread

metalliqaz

From the article:

> Strings were just a sequence of bytes in Python 2, but now are Unicode by default. Much better.

whelp, not trusting this guy's judgement anymore

View on HN

Python also has a bytes type that’s the same as python 2 strings. Why would you want your strings to not be Unicode?

Obviously nobody cares what I think, so perhaps you would prefer the words of Armin Ronacher, the creator of Flask.

http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/

I've been using python for years now, I don't know the last time I had to use `sys.stdin`. I (like most people) deal with `open`, and unicode-only file apis (or the very nice things like flask that mitsuhiko and others have built). For most people, this isn't a problem.

I've migrated literally, 10s of thousands, of files from py2 to 3. I've yet to encounter an issue with surrogateescapes or sys.stdin. Unicode issues yes, but mostly because we were previously being fast-and-loose with unicode vs. bytes, and mishandling, for example, emojis.

I certainly wouldn't trust Python to open the standard streams correctly, explicitly check the mode/encoding or reopen the descriptor.

I have a lot of respect for Armin as a smart engineer and all around good guy, but I think he was and is wrong about this one.

While the sentence is a bit weird, along the lines of "That word is made up." "All words are made up." strings ARE potentially harder to deal with in 2 than 3.

in my experience, most people want bytestrings most of the time, and the main effect of changing the str type has been to drive a lot of traffic to SO when they have encoding errors trying to deal with strings that used to "just work" in 2.7

The thing about "just works" in 2.7 is that it does ... until it doesn't, and when it doesn't, it's often way harder to resolve.

If you're only using the original ASCII (ordinals 0-127), all is well, and the py3 distinction between strings and bytes seems to get in the way. But as soon as you have a single character in your input or data which is not ASCII, a lot of things stop working and it isn't even clear why, or how to fix it -- often, the error comes from a library-within-a-library-within-a-library, which did some manipulation that (wrongly) assumed some encoding or its properties.

Maybe you're right. Maybe there are applications where this really matters.

All I can say is that I have never once had that problem since I started using Python (circa version 2.4). It has always been fine. For example, when dealing with .csv files created with MS Excel, you get strange characters because Excel uses CP1252 encoding, not ASCII. They never tripped up my app in python 2.7. It happily passed them along. Even regular expressions worked. Python3 died repeatedly until I went through and forced everything to be bytestrings.

That's funny as I had the opposite of experience - pulling non-ASCII characters into Python 2.7 caused me trouble - I ended up having to mark everything as Unicode to solve the issues.

You're reading non-Unicode into strings that expect Unicode, and since there's no way to auto-detect CP1252, you get bad results. If you want to stay in CP1252 you should clearly use bytestrings. You could convert to Unicode, I expect that:

text = file.read().decode('cp1252')

would work (haven't tried it, though).

I frequently work with CJK text, and Python 2 was a real pain. You had to do things one way to output to a terminal, but a different way to output to a pipe, and all your code had to remember to encode('utf8') every time you printed a string (unless you were using pipes, if I recall correctly). The situation is much better with Python 3, especially if you keep your files in UTF-8. Unfortunately, I think Windows is still stuck in their own world, so it might be harder to get UTF-8 by default. In Unix pretty much everything writes UTF-8 files by default, so everything has pretty much just worked for me (macOS/Linux).

> Even regular expressions worked.

Did you try doing regular expressions for non-ASCII characters? For example, given a file "latin1file" containing the string "L£", encoded in Latin-1, Python 2 will fail silently:

  $ python2 -c 'import re; s = open("latin1file").read(); print(re.search("L", s));'
  <_sre.SRE_Match object at 0x7fb992a44bf8>
  $ python2 -c 'import re; s = open("latin1file").read(); print(re.search("£", s));'
  None

Python3 forces you to deal with the problem to handle the first case, but if you do, the second case will work as well:

  $ python3 -c 'import re; s = open("latin1file").read(); print(re.search("L", s));'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/lib/python3.7/codecs.py", line 322, in decode
      (result, consumed) = self._buffer_decode(data, self.errors, final)
  UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 1: invalid start byte

  $ python3 -c 'import re; s = open("latin1file", encoding="latin1").read(); print(re.search("L", s));'
  <re.Match object; span=(0, 1), match='L'>
  $ python3 -c 'import re; s = open("latin1file", encoding="latin1").read(); print(re.search("£", s));'
  <re.Match object; span=(1, 2), match='£'>

I've never had to do that. Maybe because I live in the U.S.

I’m not quite sure about this. I think while python 3 has a tendency to blowup on IO, I still think that is preferable than more cryptic problems. Python 3 did seem to handicap working in bytes though, but I think that has improved in the last few versions with the path objects and str accessing Unicode text.

Reply on news.ycombinator.com