Comment by nneonneo | Hacker News Reader

Comment by nneonneo | Hacker News Reader

Python 3 has arguably one of the best built-in string implementations around.

In Python 2, "unicode" was a type whose codepoint width depended on the interpreter build - 2 bytes on "narrow" builds and 4 bytes on "wide" builds. Since most builds were "narrow", in practice non-BMP codepoints were a real challenge to use.

Furthermore, most languages with a "wide" string type use a fixed 2 bytes per codepoint (Java, C#, C++, ...), which wastes space when you're dealing with ASCII a lot, and is a pain to work with if you have non-BMP codepoints, since now indexing requires scanning the string from start to end.

In Python 3, with PEP 393, strings now have a flexible internal representation which can use 1, 2, or 4 bytes per codepoint depending on the largest codepoint in the string. This saves both space and processing time (in most common situations) since common string operations like indexing and slicing are constant time. This representation also allows it to scale from ASCII to astral codepoints with ease.

Python 3 is worth the switch. Correct, strict separation of 'bytes' (bucket of octets) and 'str' (sequence of codepoints) is really the only way to preserve sanity and interoperability with today's encoding-rife reality.

I would say Swift has the best implementation. It actually conceptually treats strings as a group of grapheme clusters (the Character type is a grapheme cluster). This is more in line with how humans conceptually segment strings, and how most application logic should segment a string.

> since now indexing requires scanning the string from start to end.

This is not a good thing. O(1) code point indexing is a completely useless operation. Code points have no intrinsic meaning. They do not map to any natural language concept. They are just a tool to encode strings. The only time code points matter are when writing algorithms defined by the unicode spec, like casefolding. These operations usually require iteration anyway.

If you're regularly indexing and slicing strings by code points, your code is likely broken when it comes across decomposed accents in European languages, most Asian languages, and emoji.

FWIW you can have O(1) slice indexing in UTF8 as well; you just need to use byte position as your indexing type (and handle the error case when it's not aligned). Rust does this. It also has methods for working with char indices (and a .char_indices() iterator that lets you juggle the two), but byte indexing works. Generally these indices come from other parts of the application so you can design it to not have errors.

I end up having to explain this often, and have been meaning to write about this for a long time, and I finally did:

http://manishearth.github.io/blog/2017/01/14/stop-ascribing-...

I'm not disagreeing but the situation is a bit more complex. For example ISO 10646 ("Unicode") has multi-code point sequences denoting still a single character (such as variation sequences, which are actually used in standard HTML entities).

Moreover, there are languages/scripts challenging the notion that a particular byte-unit corresponds to a single character.

Treating strings as byte sequences (with optional UTF-8 interpretation/checking, uppercase/lowercase conversion if the concept even applies, trim functions based on all Unicode spaces not just those characters in US-ASCII etc.) is entirely a defensible choice for a programming language.

I was sloppy in using the word "character" - strings are really sequences of codepoints. I've edited my post to reflect this. Of course, multiple codepoints may be needed to construct a single character, and that's by design.

But has Python 3 Unicode-aware string-length rather than byte-length functions (+ substring, case-folding, character classification functions etc.) for dealing with this? If it hasn't then nothing really is achieved for the general case by switching to 4-byte or any other fixed-byte unit, including the case of dealing with HTML 5 text.

See also perlgeek's question.

Yes, Python 3 does have Unicode-aware functions, as demonstrated in the post (via slices).

Go and Rust also have very good Unicode support, in addition to Swift and (apparently) Perl 6 as mentioned above.

> For example ISO 10646 ("Unicode") has multi-code point sequences denoting still a single character

I only know two programming languages that deal correctly with that: Swift and Perl 6. If you know more, please tell me.

Rust separates this out; the standard library gives you bytes and Unicode Scalar Values. Grapheme cluster stuff is in a package on Cargo, maintained by the Servo team.

You can iterate over Grapheme clusters using the standard library: https://doc.rust-lang.org/1.3.0/std/str/struct.Graphemes.htm...

Those are unstable docs for 1.3; they don't exist in today's Rust.

Sounds like Rust follows in the tradition of C. Like you now always hear about hos C has no string type, just a bunch of char, will we hear in the future that, alas, Rust has no native support for grapheme clustering?

The issue is, there's multiple ways to do the clustering, in my understanding. We picked the most core thing to be our most core thing. Text is hard.

Oh I get it didn't want to come across as flippant. Indeed text is hard.

If you rely on libraries, there are ICU libraries for C and Java too, see for example http://userguide.icu-project.org/boundaryanalysis

Java hasn't used 2 bytes per code point since many releases ago. It does unfortunately use 2 bytes per char, but the standard library makes it reasonably easy to deal with cases where you need 2 chars for a code point.

Reply on news.ycombinator.com