Yes it is possibly to laboriously add code to check and encode/decode text, that might sometime be text and sometimes binary pretending to be text pretending to be binary. But why go through that pain when you could just use Python3 where things are clear and generally actually work.
It's also very intuitive, which is great for teaching - I've always had a hard time explaining the difference between "strings" and "unicode" objects in Python 2, but the clear boundary makes it straight-forward in Python 3.
But, personally, if you're teaching Python to a class, whether it's 2.x or 3.x. You should probably stick to using some variation of string.upper() == string.upper() to explain simple comparisons, due to simplicity.
Okay: You can't. You'll need to convert both strings to the same case, and then compare those. In the majority of cases, you can do:
return string_a.lower() == string_b.lower()
But sometimes that won't work, because converting case isn't always easy. And that's a unicode issue, not a python issue. It's not even completely a unicode issue, just a "writing can be very weird" issue.
For example, "ß" (U+00DF) is a character that doesn't/shouldn't have an uppercase form. Unicode does support it as U+1E9E, but all the examples I know of will convert it to "SS" ([U+0053, U+0053]) instead.
Try this in a javascript console, for example.
> "ß".toUpperCase().toLowerCase() === "ß".toLowerCase()
false
> "ß".toUpperCase() === "SS".toUpperCase()
true
> "ß".toLowerCase() === "SS".toLowerCase()
false
> "ß".length
1
> "ß".toLowerCase().length
1
> "ß".toUpperCase().length
2
> "ß".toUpperCase().toLowerCase()
"ss"
Also, try ctrl+f'ing for "ss" on the Wiki page for ß [0] and see what it matches. Or ctrl+f this page for "ß" and it will match every "ss". I think it's pretty neat.[0] https://en.wikipedia.org/wiki/%C3%9F
(edit: You might already know everything in this comment, but I started writing and it ended up being a cool exercise for myself, so I figured I'd submit).
Absolutely! I think there's a lot of baggage in computing because it happened to emerge in places where language could mostly be written in simple characters with surprisingly simple rules. When exported internationally suddenly you deal with "oddities" that are really rather the norm in other languages.
In French for example is customary NOT to put accents on capital letters, even though it's possible. But then again sometime people do it anyway - it's not a hard rule. So often converting to lowercase may require contextual understanding because some words are written the same but with different accents, and would capitalize to the same string. Which word was meant would only be evident from the context of the sentence.
I suspect that if computing had rather developed somewhere where these things were commonplace we would have much better handling of it today.
For the same reason if you need to write code that should work on Py2 and Py3, it's much easier to write it on Py3 and then add backward compatibility through "from __future__ import ..." and the six module.
It's much less clunky than six. Instead of manually adding the compatibility layers to your code, it automates the process, which results in cleaner code. It can even translate (some) Python 2 modules on-the-fly.
Many projects switched to it from six.
Did you know, that there is an operating system, where a simple loop that reads lines from file and prints them to the console will end up with errors? (that being Windows and some European code pages, where console is cp85x, filesystem read and write cp125x).
Well, and that's before the user tries to redirect stdout to file, in which case it will end up with error too, because that is going to be 7-bit ascii.
Anyway, Python3 does the right thing with defining the encoding at the edges and everything works right for the user out of the box.
Either way, if you're not dealing with differently encoded character data in Python 2.x as noted above, you're causing yourself some unnecessary headaches.
Compared to that, Python3 was like gift from heaven.
But seriously, all what python (and as a matter of the fact majority of languages) is placing a clear distinction between a byte and a character. This forces developers to keep in mind what they are dealing with and reduces number of bugs.
The truth is that if you treat everything as bytes, chances are that your program has bugs. Actually good test is to convert your existing Py2 code to Py3, if you have encoding/decoding errors then your code was buggy.
No, they make a clear distinction between bytes and code points. This is the wrong approach as code points are not characters. People are going to think that everything is fine now that they're using code points and the unicode type when they should be using a unicode library for text manipulation.
https://docs.python.org/3.3/whatsnew/3.3.html#pep-393-flexib...
Glancing at Julia, I'm not sure "invalid character index" is really a great thing to have to handle when dealing with text. It does seem that systems for dealing with text will eventually offer various apis beyond just a somewhat disfunctional character api (which I think is a fair description of str in Python 3).
I happen to know this fact. Are you really asking that everyone that comments on a language acknowledge all the possible implementation details? I at no point complained about the memory required to hold strings in CPython so I can't really understand why you would bring this up.
It's non-BMP characters that trigger the conversion to 4 bytes per codepoint. Unfortunately Emoji are non-BMP and are super common.
Non-BMP characters in 2.x-3.2 give you implementation-specific behavior. It can vary from one system to the next. Some string methods fail on "narrow builds" of Python (the ones that store their characters in UCS-2), while others return different results than they would on a "wide build".
And I don't want to be forced to distinguish between strings and bytes. To me they are one and the same as I use UTF-8 for everything. Not once in my code do I need to encode/decode anything.
As for regular expressions, they work perfectly if your delimiters are in the ASCII range. If you're trying to match a single character then they don't work for Unicode even when using Python3's unicode type. The reason is that code points are not characters.
For regexes, \w matches any unicode word character (unless ASCII mode is on), just use it on strings. If you use it on bytes, then yes, you can only match bytes.
And this perfectly illustrates what I posted in another comment elsewhere. Thinking that doing operations on code points (indexing and length) is the proper thing to do. You need to use a unicode library to work on graphemes and take into account grapheme clusters.
> For regexes, \w matches any unicode word character
From what I've read, \w just matches code points. I did find that the https://pypi.python.org/pypi/regex module supports \X which matches graphemes as per the unicode spec http://www.unicode.org/reports/tr29/.
I've always wanted to know how things are done internally. Makes me a better programmer. What I said about all this extra conversion does to performance is correct, especially when scaling up.
And my choice to use UTF-8 internally is a valid way to do things. I make sure that what comes in from the outside world is UTF-8 (either by rejecting or converting) and then don't have to worry about it from that point on. The bulk of my code deals with outbound data so no need for me to do any encoding as it's already UTF-8.