Comment by mjevans | Hacker News Reader

Comment by mjevans | Hacker News Reader

Parent Full thread

Python 3 has some architectural decisions that make working with data difficult; mostly ///my opinion/// is that it completely botched Unicode handling.

The idea of data being encoded should be an extremely high level idea, with foundations that are supported by the internal libraries and maybe even compiler.

I feel that golang has a much more data-structure friendly view of Unicode.

    * The ancient paradigm of bytes remains
    * Strings are immutable sequences of bytes; but are trivial to cast back and forth.
    * 'runes' (distinct Unicode code points) also exist

However to actually validate or do anything fancy to a Unicode sequence a programmer would need libraries from https://godoc.org/golang.org/x/text which shows how truly complex /proper/ handling of Unicode data is; instead of just the simple copy / append / search (is this inside of / a match to it) operations that make up 99% of my handling of Unicode data.

What about Python3's unicode handling is botched?

It optimized for O(1) indexing to Unicode scalar values instead of optimizing for more useful things. Also, even though the API semantics are UTF-32, the storage may be USC-2 or Latin-1, so there's a surprise performance effect when a string gets it first non-Latin-1 character or its first non-BMP character. So much for O(1) behavior.

The best in-memory representation of Unicode is UTF-8. No alignment issues, ingest "decode" is mere validation and output "encode" step is a no-op. Space requirements are favorable to common computer language syntaxes. Byte-wise lexical sort matches code point-wise lexical sort.

However, since "narrow builds" of Python 2 already used UTF-16 which allowed for things like bridging strings with Cocoa, Python would have done well to stick to UTF-16 and get rid of the "wide builds".

Not having both "narrow" (UTF-16, Python 2 on Mac) and "wide" (UTF-32, Python 2 on Debian) is an improvement, though.

Python strings are immutable. The only time a string can get its first non-Latin-1 character is at construction time, which is necessarily O(n).

Have you worked in Python 2 unicode?? Over half of the work and debugging the search engine service management team I was on was fixing unicode document injesting.

But I agree, making unicode part of the language is very dumb. I just wish they would give up as it has never been good.

Reply on news.ycombinator.com