Hacker News Reader

Comment by alextgordon | Hacker News Reader

It seems the most obvious solution is to store strings in a standard encoding (say UTF-8) and to always convert strings to it at the time of their creation. Is there a technical reason why Ruby doesn't do this?

View on HN

The technical reason is that it's a stupid idea. Not all encodings can be safely round-tripped through UTF-8, which means you can end up losing some data. (Consider http://homepage1.nifty.com/nomenclator/perl/ShiftJIS-CP932-M... as a quick example: "Actually, 7915 characters in CP-932 must be mapped to 7517 characters in Unicode. There are 398 non-round-trip mappings.")

Loss of text data is bad.

That page says there are duplicates in CP-932 because of different vendor variants. If those characters are otherwise entirely identical, calling it a single encoding seems wrong. Wouldn't you just have a CP-932-IBM and a CP-932-NEC encoding?

Because someone unified two similar encodings into CP-932/ShiftJIS. That someone who unified them wasn't IBM or NEC (both of whom made competing systems and had made different encoding choices, but whose choices were mostly compatible).

Rules for dealing with legacy encodings: 1. They make no sense. 2. If you think they make sense, remember that you weren't there so refer to rule 1.

General question: Why didn't the UTF-8 boys and girls make it safe in the first place? This doesn't sound like rocket science. "This character maps to that character, this character to that one." I don't understand how we have unicode snowmen, but we can't safely round trip characters.

There isn't a good idea that a standards body hasn't fucked up. Maybe it's because these problems are harder than they seem. Or maybe they're just plain screwed up. Or maybe it's Adobe.

Some of it has to do with Han Unification (http://en.wikipedia.org/wiki/Han_unification).

Mostly, though, it's because some of these characters are overloaded. If you've got a Windows system, go into the DOS window and type "chcp 932" (you may need the Japanese language files installed). When you type '\', you'll get '¥' (making "C:\Program Files\" look like "C:¥Program Files¥").

In the systems where what become CP932 were first used, the backslash wasn't necessary in Japanese, so that character point was used to encode the yen symbol. Other systems used the backslash, so it was encoded as a different point. When JIS unified the existing Japanese code pages, it couldn't very well go back in time to change all that old data, so it merged the two encodings on many things. So, there's only one Unicode codepoint for the yen glyph ¥, but in this one encoding there's two different characters for it.

This is the most blatant example of a problem with Unicode transcoding, but as far as I know, it's not the only one.

See http://www.mail-archive.com/linux-utf8@nl.linux.org/msg02337... for what could be done, but probably won't.

Are these encodings common enough that they're worth supporting in the basic String class? Surely another class could be provided to handle these edge cases, they don't have to be natively handled.

There's no need for any data loss to occur - the String class would merely not support converting from non round-trippable encodings.

If they're not natively handled, you can't regex them. If they're not natively handled, you can't convert any numbers that might be in them to numeric values.

Yes, they're common enough (especially in Japan) and encodings have to be baked deeply in if you really want to use everything that a Rubyist expects to be able to use.

Practically speaking, what data are you talking about? Which characters don't map? The mapping here looks pretty complete, except for "DBCS LEAD BYTE". Is that a problem? Other non-round trip mappings look like it's because different vendors have different meanings for that character. That's not Unicode's fault.

http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOW...

If there's ~398 byte combinations that can be translated only one way, but not the other, you're still losing data. (The big one that always gets pointed out? '\' and '¥'.)

"(The big one that always gets pointed out? '\' and '¥’.)”

The point there though is that Japanese DON’T want \ and ¥ to map properly. They want \ and ¥ to be considered the same. So Unicode isn’t losing information, it’s forcing a distinction the Japanese don’t want to be able to make.

Java stores all strings in UTF-16. It probably seemed like a good idea at the time but has led to problems later. There is now a huge body of broken code out there which assumes that each 16-bit char value represents a complete and separate Unicode character.

Um. Wasn't Java's original choice UCS-2 (same as NTFS originally)? UTF-16 has surrogate characters and the length will be one or two 16-bit values depending on whether it's a surrogate. UCS-2 was a fixed 16-bit character size, long since deemed a mistake.

You're right. Java originally used UCS-2 and later switched to UTF-16. http://java.sun.com/docs/books/jls/first_edition/html/3.doc.... The problem is that dealing with surrogate pairs is so awkward that most developers don't even bother.

> Is there a technical reason why Ruby doesn't do this?

Han Unification. Gory details here: http://en.wikipedia.org/wiki/Han_unification

I'm not a Ruby-head, so maybe I don't have the necessary context to understand what's going on, but: As far as I'm concerned, the only time I care about encodings is when I'm trying to do I/O. I'll always have to choose an encoding when engaging in I/O, explicitly or implicitly. While I'm shuffling strings around in memory, I expect it to behave like a sequence of unicode code points. If the implementation wants to allow multiple implementations for performance reasons (e.g. because reading UTF-8 from file, converting it to UTF-16, processing it, converting it back to UTF-8 to send it over the network is inefficient) then that's fine by me. I'll still want to tell it what encoding that byte stream from which I'm writing is, and what that stream I'm writing to expects. If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings. That was a bad idea in C and with the complexity that unicode adds, this is an even worse idea nowadays.

If I've understood the rant correctly, Ruby (1.9?) confuses byte streams with strings.

My understanding was just the opposite: now that strings are associated with encodings, he can no longer assume that a1 + a2 results in a string with the same encoding as a1 and a2, since a1 and a2 can have different encodings.

The implication is that the programmer must know what is used internally, though, because things will go horribly wrong otherwise. That's a sign of a very leaky abstraction - if you need to know the format and behaviour of the underlying byte stream, you'd probably be better off using the byte stream directly, the abstraction is just introducing uncertainty where it should be isolating from such concerns.

It's also supposed to be an incorrect implication for Ruby most of the time. It should be possible to do a single encoding trivially (e.g., if all of your data is shift_jis or UTF-8), but rather than trying to (badly, and usually unsuccessfully) hide the encoding difficulties from the programmer if you've got mixed encodings, Ruby has chosen to be a little more up front about encodings.

The reality may be a bit differently, but I recall seeing an email message from Matz on ruby-core last year suggesting that it was supposed to be trivially easy to work with one encoding (specifically mentioning UTF-8, but implying others).

I also thought the exact same thing, but I know that encoding stuff is so complex that I may be missing something. For what its worth, I believe Cocoa does things this way: a string is an abstract object of unicode code points, when its time to read or write you must choose an encoding.

The Java and .NET infrastructures make that choice for you too. I guess I wasn't there when Java first got started, it's possible that there was a backlash[1] against this decision by the people who are using encodings that aren't subsets of unicode. I don't get to see many "enterprise" systems, but I get the impression those applications where it's important to use some kind of not-unicode are sufficiently rare that they're best served by specialty libraries. You can't make everyone happy with one solution, but if a simple solution works for 99%, then I'd call that a roaring success.

[1] a backlash other than the OMG-2-bytes-per-ASCII-character hysteria, which is irrelevant for these purposes, they could just as well have chosen UTF-8

Is there a technical reason why Ruby doesn't do this?

More likely it's political. Matz (the creator of Ruby), and many of its early contributors, are said not to like Unicode.

That doesn't jive with what I know of Matz. Japan (and much of Asia) adopted Unicode much slower because both UTF-8 and UTF-16 are less efficient than many of the native encodings for representing much of what they need; there was definitely some cultural tone-deafness in the Unicode community when it did Han unification (which may be a good idea, but could have been handled better).

Matz (and the people he works with who use Ruby to get their jobs done) needs access to data that's Not Unicode. Painfully Not Unicode as in it doesn't necessarily round-trip.

The performance is better if you use an encoding with a fixed byte-length character-set.

No it isn't.

And performance... My non-utf8 1 GB file whose number of lines I need to count thanks Ruby 1.9 for not uselessly and forcefully making an in memory utf8 copy of it.

Reply on news.ycombinator.com