Hacker News Reader

Comment by 0x0 | Hacker News Reader

The most hilarious issue with -Dfile.encoding is that if it set to, for example, UTF-8, then it is literally impossible to open files with names encoded in, say, ISO-8859-1, which can happen on a Linux system where users use different LC_CTYPE.

You can do a "dir.listFiles()" and iterate it, and find that some of the entries are impossible to open because there is no way to represent the ISO-8859-1 bytes that make up the filename in a String object, and therefore no way to give the correct file name to the java io classes for opening.

View on HN

That gives more weight to the argument that character encodings should be entirely a "presentation-layer" concern, and anything below that should treat strings as opaque byte sequences. There should also be a way to bypass that "presentation layer" transformation upon input.

Absolutely, there's quite a few gaffes in java (other languages have similar issues too):

* char being 16bit and thus can't hold a modern unicode character

* lots and lots of string and byte stream methods that have variants without explicit charset parameters, making it very easy to accidentally have a hidden dependency on the implicit default charset (aka -Dfile.encoding)

* Using strings instead of byte[] for filename values

* Having to deal with UnsupportedEncodingExceptions as checked exceptions whenever referencing the UTF8 encoding

The reason for char being 16 bits is that the JLS predates Unicode 2.0 (1996), which added 32 bit codepoints. At that time, UCS-2 was widely believed to be sufficient to represent anything "international."

Indeed, this is exactly what the Linux kernel does; it treats filenames like a binary blob, with the exception that the bytes 0x2f (ASCII '/') and 0x00 (ASCII NUL) are not accepted (regardless of where they are in the byte string).

> it treats filenames like a binary blob

Which can result in files with seemingly identical names when one UTF-8-encoded file name uses combining characters and the other one uses precomposed ones. Not to mention the fact that e.g. Å is contained in Unicode as both "latin capital letter A with ring above" and "Angstrom sign".

After you've solved encoding, next comes Unicode normalization ;-)

It is not a big deal as long as seemingly identical file names are treated by OS as different.

Unicode normalization is not the only problem here, e.g. Latin 'a', 'e', 'T' are exactly the same as Cyrillic 'а', 'е', 'Т' in most fonts which makes it possible for two files to have seemingly same names even in some 8-bit encodings.

Even Latin 'I' and 'l' and the digit '1' are visually indistinguishable in some fonts! So are trailing spaces and different numbers of spaces. This is such a pervasive problem that maybe we can just give up and expect users to get used to it.

Wow, so characters like U+022F (ȯ) and U+042F (Cyrillic letter Я) U+062F (Arabic letter د) are not allowed but nearly everything else is? Some of those are letters used in actual languages. That's sure to make people scratch their heads.

When encoding those characters in UTF-8, you will never end up with 0x2F as a byte. On of the properties of UTF-8 is that bytes with the high bit not set (e.g. 0x0 to 0x7e) never appear unless the are representing the 0 - 127 codepoints.

    codepoint -> encoding in UTF-8

    U+022F    -> C8 AF  
    U+042F    -> D0 AF  
    U+062F    -> D8 AF

Remember, UTF-8 is self-synchronising: when you pick up at a random point within a stream, there is no ambiguity as to whether you are in the middle of a sequence or not. Valid lower codepoints appearing in the encoding of higher codepoints would violate this property.

If you encode them in UTF-8, they are allowed. In UTF-8, to represent a given code point, only the first code unit can be in the range 0x00 to 0x7F. So if you're reading UTF-8 content and find a byte with value 0x00-0x7F, you can be sure it is a 1-byte long code unit sequence representing the code points U+0000 to U+007F.

> which can happen on a Linux system where users use different LC_CTYPE.

If only it were that simple. POSIX paths are encoding-less bags of bytes, you can literally find anything in them aside from NUL, LC_CTYPE isn't even a consideration.

Reply on news.ycombinator.com