Hacker News Reader

Comment by gnud | Hacker News Reader

This idea that some developers don't need to deal with unicode, really needs to die.

If you're creating a web app and don't support unicode input, you're doing it wrong. You probably can't accurately store the names of your customers, for starters.

View on HN

OP here. I teach Python to Fortune 100 companies, more or less every day.

You're totally right that people who deal with outside-world data need Unicode. Anything that deals with users MUST deal with Unicode.

But a huge proportion of the people I work with -- in Israel, China, and Europe -- couldn't care less about Unicode. That's because they're processing logfiles, dealing with test automation and system administration. For them, Unicode is important a very small proportion of the time.

So I stand by my assertion that a small proportion of developers need to worry about Unicode. But I also agree with you that Unicode is absolutely crucial in the modern world, and that a language lacking good Unicode support isn't going to get very far.

> That's because they're processing logfiles, dealing with test automation and system administration ... So I stand by my assertion that a small proportion of developers need to worry about Unicode.

You come to this conclusion based on your dealings with a small subset of developers within a small subset of companies?

I think many other developers in the fortune 500 companies outside of the subset you deal with would laugh at that conclusion, then go back to developing apps that deal with people/locales all over the world.

Fair enough. My sample is from lots of companies, but a small subset of tasks there.

This also describes my entire team. And the team before that. There are at least as many developers out there who don't write user facing code as those who do.

When I do run across Unicode, it's just a matter of passing it through, which is usually pretty easy to do.

Just because your users are dealing with logfiles, their users should still be able to use č ƒ ø or whatever unicode character in the filename or url - which will consequently end up in a log file.

Most logs are actually bytestrings. If they include user-controlled strings, they will likely have to be parsed as binary blobs to accommodate that.

Many logs are localised - and contain unicode.

There's a difference though. Does the log format itself in some unicode encoding, or does it just embed some utf-X encoded strings? If you just embed some user input and assume it will decode properly, you're likely to run into issues at some point when someone provides explicitly-broken encoding.

True story: I once broke build automation by including non-ASCII text in the commit message. Just because something is internal doesn't mean that it won't have Unicode.

Pro tip: Use non-ascii characters in your test data. Instead of John Smith, try Helmut Müller. This will make sure your code is unicode safe, and also make sure that external dependencies (hey, our spreadsheet library silently fails when a ü is in the file name!) also work correctly with unicode.

German Umlauts are not a good test vector, because various SBCS have them as well (latin-1 and others), and can slip through systems not correctly processing Unicode.

Use CJK instead, eg. 나윤선, which will just explode into an obvious gooey mess if the processing pipeline isn't handling Unicode correctly somewhere.

I use a site like http://lingojam.com/FancyTextGenerator to generate 𝖚𝖘𝖊𝖋𝖚𝖑 𝖙𝖊𝖘𝖙 𝖉𝖆𝖙𝖆.

The only downside is that it has taught me that my favourite debugger is not unicode safe :(

Also, you can litter your test data with all the fancy emoji you can type in a modern OS these days, those are almost always instantly recognizable when correct versus incorrect and test a fun number of things given:

1) Most (but not all) emoji exist way out in the Astral Plane (a good stress test for UTF-8/UTF-16 handling and little/big endian issues)

2) Emoji can contain "exotic" sequences like Zero Width Joiners (this can be very important to avoid codepoint counting issues because a ZWJ sequence of codepoints is a very different emoji if the codepoints are incorrectly reversed or spliced)

3) Emoji can test your font display stack for inefficiencies in system font fallback (and support for modern typography in general)

I like to use "Tokyo" in kanji, since I can remember what the characters look like: 東京

Is that a good test?

Many programmers don't need to deal with Unicode simply because they aren't dealing with natural language text in the first place. Programs that need to support Unicode input are precisely the ones that have natural language text input.

I do primarily numerical work with a little bit of embedded mixed in, it's been closing in on three years since I encountered anything involving non-ascii characters.

yes, in webapps you almost always import from future unicode literals in py2 apps.

Reply on news.ycombinator.com