Comment by jcranmer | Hacker News Reader

Comment by jcranmer | Hacker News Reader

Parent Full thread

> There's not really a good reason to support UTF-8 over UTF-16

Two big reasons:

1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII.

2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examples of this.

> Write your programs/whatevers allowing your users to be aware of and configure encodings.

Internally, your program should be using UTF-8 (or UTF-16 if you have to for legacy reasons), and you should convert from non-Unicode charsets as soon as possible. But if you're emitting stuff... you should try hard to make sure that UTF-8 is the only output charset you have to support. Letting people select non-UTF-8 charsets for output adds lots of complication (now you have to have error paths for characters that can't be emitted), and you need to have strong justification for why your code needs that complication.

Every program that purports to support Unicode should be tested with a bunch of emoticons.

Do you mean emoji? I don't see what the issue would be with [{}:();P\[\],.<>/~-_+=XD]

Yes, that's what I meant. I knew I was using the wrong word but couldn't remember the right one.

> 1. All legal ASCII text is UTF-8. That means upgrading ASCII to UTF-8 to support i18n doesn't require you to convert all your files that were in ASCII.

Eh, realistically if you're doing this, you should be validating it like converting from one encoding to another anyway. I get that people won't and haven't, but that's because UTF-8 has this anti-feature where ASCII is compatible with it, and that's led to a lot of problems.

> 2. UTF-16 gives people the mistaken impression that characters are fixed-width instead of variable-width, and this causes things to break horribly on non-BMP data. I've seen amusing examples of this.

This is one of those problems, and it's way worse with UTF-8 because it encodes ASCII the same way ASCII does. It's let programmers stay naive about this stuff for... decades?

> Internally, your program should be using UTF-8 (or UTF-16 if you have to for legacy reasons), and you should convert from non-Unicode charsets as soon as possible.

There are all kinds of reasons to not use UTF-8. tialaramex pointed out one above. "UTF-8 everywhere" is simply unrealistic, and it forces a lot of applications to be slower, or to take on unnecessary complexity. Maybe it's worth it to "never have to think about encodings again", but that's pretty hard to verify and there's no way it happens in our lifetimes anyway.

> and you need to have strong justification for why your code needs that complication.

Yeah see, I strongly disagree with this. I'll choose whatever encoding I like, thanks. Maybe you don't mean to be super prescriptive here, but I think a little more consideration by UTF-8 advocates wouldn't hurt.

> I'll choose whatever encoding I like, thanks.

If everyone chooses whatever encoding they like, then the charset being used has to be encoded somewhere. The problem is, there are lots of places where charset isn't encoded (such as your filesystem). That this is a problem can be missed, because almost all charsets are a strict superset of ASCII (UTF-{7,16} are the only such charsets to be found in the top 99.99% of usage), so it's only when you try your first non-ASCII characters that problems emerge.

Unicode has its share of issues, but at this point, Unicode is the standard for dealing with text, and all i18n-aware code is going to be built on Unicode internally. The only safe way to handle text that has even the remotest change of being i18n-aware is to work with charsets that support all of Unicode, and given its compatibility with ASCII, UTF-8 is the most reasonable one to pick.

If you want to insist on using KOI-8, or ISO-2022-JP, or ISO-8859-1, you're implicitly saying "fuck you" to 2/3 of the world's population since you can't support tasks as basic as "let me write my name" for them.

> If everyone chooses whatever encoding they like, then the charset being used has to be encoded somewhere.

This is gonna be the case for the foreseeable future, as you point out. Settling on one encoding only fixes this like, 100 years from now. I'd prefer to build encoding-aware software that solves this problem now.

> given its compatibility with ASCII, UTF-8 is the most reasonable one to pick

This only makes sense of your system is ASCII in the first place, and if you can't build encoding-aware software. I think we can both agree that's essentially legacy ASCII software, so you don't get to choose anything anyway. And any system that interacts with it should be encoding-aware and still validate the encoding anyway, as though it might be BIG5 or whatever. Assuming ASCII/UTF-8 is a bad idea, always and forever.

> If you want to insist on using KOI-8, or ISO-2022-JP, or ISO-8859-1, you're implicitly saying "fuck you" to 2/3 of the world's population since you can't support tasks as basic as "let me write my name" for them.

I'm not obligated to write software for every possible user at every point in time. It's perfectly acceptable for me to say, "I'm writing this program for my 1 friend who speaks Spanish" and have that be my requirements. But if I were to write software that had a hope of being broadly useful, UTF-8 everywhere doesn't get me there. I'd have to build it to be encoding-aware, and let my users configure the encoding(s) it uses.

> But if I were to write software that had a hope of being broadly useful, UTF-8 everywhere doesn't get me there.

Actually, it does.

Right now, in 2020, if you're writing a new programming language, you can insist that the input files must be valid UTF-8 or it's a compiler error. If you're writing a localization tool, you can insist that the localization files be valid UTF-8 or it's an error. Even if you're writing a compiler for an existing language (e.g., C), it would not be unreasonable to say that the source file must be valid UTF-8 or it's an error--and let those not using UTF-8 right now handle it by converting their source code to use UTF-8. And this has been the case for a decade or so.

That's the point of UTF-8 everywhere: if you don't have legacy concerns [someone actively using a non-ASCII, non-UTF-8 charset that you have to support], force UTF-8 and be done with it. And if you do have legacy concerns, try to push people to using UTF-8 anyways (e.g., default to UTF-8).

I can't insist that other systems send your program UTF-8, or that the users' OS use UTF-8 for filenames and file contents, or that data in databases uses UTF-8, or that the UTF-8 you might get is always valid. The end result of all these things you're raising is "you can't assume, you have to check always, UTF-8 everywhere buys you nothing". Even if we did somehow get there, you'd still have to validate it.

Reply on news.ycombinator.com