Some Thoughts on Unicode in Perl 6
rdstar.wordpress.com
rdstar.wordpress.com
http://stackoverflow.com/questions/6162484/why-does-modern-p...
http://dheeb.files.wordpress.com/2011/07/gbu.pdf
(The official URL is: http://training.perl.com/OSCON2011/index.html but it's currently timing out for me)
It do feel like getting information on the construction method of the whips which will be used on your back sometime in the future. :-) :-(
And the first comment (which i did go through for my ventures into perl unicode) is nothing but prove how broken it is. So maybe you are just being sarcastic and i didn't get it.. ;)
First of all you don't NEED most of the stuff shown there. The boilerplate at the bottom is there to cover EVERY POSSIBLE THING anyone would possibly ever want to do with unicode in Perl and because tchrist thinks it's funny to do such things. If all you want is to read a unicode file, change its contents, then write it back to hdd without breaking encoding, you don't need more than:
use utf8;
use IO::All -utf8;
[citation needed]
Jokes aside :) if you look at the 2011 OSCon presentation pdf, you'll see it's vastly outdated (and imho the default internal encoding table is just imprecise)
OTOH I don't use unicode regexps very often, so my POV is biased (I'm sure that on this front perl5 can actually be better than virtually everything else)
For a rather complex multithreaded client/server program that reads and writes difference sources (http, webdav, ldap, file, terminal) with a lot of dependencies it really was a nightmare, trust me. Maybe i didn't get some obvious global "do that unicode stuff right" switch and i am just too dumb. But even then, i've never had such problems in _any_ other language.
Interestingly i did need to write a very similar tool one year later in python and i had much less problems. And that was python 2.x which is always said to have bad unicode support. Maybe it's just because in python (afaik) you only have to take care of some str() or unicode() semantics and not about a gazillion of use statements, open() parameters and whatnot.
Maybe Perl5 has perfect unicode support but how to use and achieve that is a freaking nightmare in a non-trivial program.
When dealing with any input/output, find out whether you read/write bytes from/to it or characters, and depending on what you find, decode or don't.
I'm very sure that the only reason you had trouble with your first tool was a lack of knowledge and thus a lack of pervasive diligence, leading to you leaving some inputs/outputs unhandled or even worse, handled double.
Then you ended up doing that stuff more correct on your second try, since python asks you to do the same things.
All the use statements in Perl do is make it so you actually don't need to do that because it makes certain operations work in incode mode by default.
In short: Please try to actively separate your own past shortcomings and growth as a developer from judgements on the capability of a language.
So, no, i'm not convinced perl has good unicode support in terms of actually supporting the programmer. Mind you, that was 2 or 3 years ago, so maybe the situation is different now. To me, it was just a stupid task of creating new tests, finding a warning and then trying to figure out what the fix is. Sometimes the fix broke other parts or libraries and then you go to fix #2 (because of course there are a gazillion ways in Perl to do a single task). To me it was really not a very good experience. And really, just look at that stackoverflow article. That's a consistent and good unicode support? No. Maybe the current perl is different, though (i think the project ran 5.12 or 5.14).
Also, as to your question about the SO post. You claim:
"The fix for them was always one of those things from the stackoverflow list."
I find myself thoroughly baffled by that, since over half that post busies itself with promoting warnings into fatal errors that end the program with stacktraces.
Honestly, i'd love to see your warnings-spitting tool, just so i could straighten it out some.
* How do you compare strings in Ruby so that an "o with umlaut" as a single character can be compared against a "o" + "umlaut" characters. Technically they are both equivalent, but if you're doing comparisons by single characters (or bytes!), then you'll run into issues.
* How do you use a regular expression in Python to match against o and "o with umlaut" as equivalent?
Yes I know I'll be downvoted to zero karma.