Okay.
> it's resulted in a simpler, easier-to-use language.
Does `print(len("ẅ"))`[0] still produce a value (2) that is neither the number of characters (1) nor the number of bytes (3)?
0: "print\x28len\x28\x22\x77\xCC\x88\x22\x29\x29"
~$ python3
Python 3.7.3 (default, Mar 27 2019, 09:23:15)
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> len("ẅ")
1
>>> "ẅ".encode('utf32')
b'\xff\xfe\x00\x00\x85\x1e\x00\x00'
Python 3.7.3 (default, Mar 27 2019, 09:23:32)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> len("ẅ")
2
EDIT: Your "ẅ" doesn't seem to be the same as the OP's "ẅ", although they look the same at first glance.
>>> "ẅ".encode('utf-8')
b'w\xcc\x88'
>>> "ẅ".encode('utf-8')
b'\xe1\xba\x85'
>>> import unicodedata
>>> w1 = "ẅ"
>>> w2 = "ẅ"
>>> unicodedata.name(w1)
'LATIN SMALL LETTER W WITH DIAERESIS'
>>> unicodedata.name(w2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: name() argument 1 must be a unicode character, not str
>>> unicodedata.name(w2[0])
'LATIN SMALL LETTER W'
>>> unicodedata.name(w2[1])
'COMBINING DIAERESIS'
> the number of characters should be in a string isn't always clear-cut.
This is why I use examples from latin-with-diacritics, where there is no ambiguity in character segmentation.
Still, to the original point, I think this is more of a criticism of Unicode than of Python. It seems to me that the answer is to not use combining diacritics, and that Unicode shouldn't include those.
True, although it's more specifically a criticism of Python for using Unicode, where these kinds of warts are pervasive. See also "\xC7\xB1" (U+01F1 "DZ") which is two bytes, one code point, and two characters with no correspondence to those bytes.
> the answer is to not use combining diacritics
This doesn't actually work, sadly, because you can't represent eg "f̈"[0] without some means of composing arbitrary base characters with arbitrary diacritics.
0: If unicode has a added a specific NFC code point for that particular character, then that's bad example but the general point still stands.
Well, yes. To be fair, it's not like any of them make a secret of the fact that they're mistakenly counting unicode code points instead of characters.
Consider, for example, the wonderful piece of writing in the answer to this question.
https://stackoverflow.com/questions/1732348/regex-match-open...
How many characters do you suppose are in this string?
.
"TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚ N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ"
.
And what should Python tell you the length of this string is?
Unicode is complicated in some ways because the domain it is dealing with (representing all possible human written communication, basically) is complicated. Unicode is pretty ingenious. It pays to invest in learning about it, rather than assuming your "naive" conclusions are what it "should" do (and unicode's standard docs are pretty readable).
Unicode does offer an algorithm for segmenting text into "grapheme clusters", specifically "user-perceived characters." https://unicode.org/reports/tr29/
It's worth reading that document when deciding what you think the "right" thing to do with "len()" is.
The "user-perceived character segmentation" algorithm is complicated, it has a performance cost... and it's implemented in terms of the lower-level codepoint abstraction.
Dealing with codepoints is the right thing for most platforms to do, as the basic API. Codepoints are the basic API into unicode.
It's true that they ideally ought to also give you access to TR29 character segmentation. And most don't. Cause it's hard and confusing and nobody's done it I guess. It would be nice.
If you want to know "well, howe come codepoints are the basic unicode abstraction/API? Why couldn't user-perceived characters be?" Then start reading other unicode docs too, and eventually you'll understand how we got here. (For starters, a "user-perceived character" can actually be locale-dependent, what's two characters in one language may be one in another).
It is specifially a abstraction that is not useful.
> It's worth reading that document when deciding what you think the "right" thing to do with "len()" is.
Technically not - the right thing to do is return the number of characters[0] - but the character segmentation parts are worth reading when deciding how to decode UTF-8 bytes into characters in the first place, so the distinction is somewhat academic.
> a [character] can actually be locale-dependent, what's two characters in one language may be one in another
[citation needed]; ch, ij, dz, etc are not examples, but I'm admitted not exhaustively familiar with non-latin scripts[1], so I would be interested to see what other scripts do.
0: or bytes, but that's trivial
1: Which is why I hate Unicode; I'd prefer to pawn that work off on someone else and just import a library, but Unicode has ensured that all available libraries are always unusably broken.
> 0: or bytes, but that's trivial
In what encoding? The utf-8, utf-32, and utf-16 encodings of the same string are different numbers of bytes.
"\xC4\xAC" is two bytes regardless of whether you interpret it as [latin capital i + breve] or [hangul gyeoh] or [latin capital a + umlaut][not sign] ("Ĭ" / "곃" / "Ĭ").