I can't really blame NT for this, it's unfortunate timing and it remains for backwards compatibility purposes. Java and JavaScript suffer similar issues for similar reasons.
I can't really blame NT for this, it's unfortunate timing and it remains for backwards compatibility purposes. Java and JavaScript suffer similar issues for similar reasons.
UTF-8 is convenient on Unix systems since it fits into 8-bit character slots that were already in place; file systems have traditionally only forbidden the NULL byte and forward-slash, and all other characters are valid. From this fact, you can use UTF-8 in file names with ease on legacy systems, you don't need any operating system support for it.
UTF-8 is "space optimized" for ASCII text, while most extra-ASCII Latin, Cyrillic, Greek, Arabic characters need two bytes each (same as UTF-16); most of Chinese/Japanese/Korean script in the BMP requires three bytes in UTF-8, whereas you still only need two bytes in UTF-16. To go further beyond, all SMP characters (eg, most emoji) require four bytes each in both systems.
Essentially, UTF-8 is good space-wise for mostly-ASCII text. It remains on-par with UTF-16 for most western languages, and only becomes more inefficient than UTF-16 for east-Asian languages (in such regions, UTF-16 is already dominant).
Seriously, when was the last time where you both cared about saving a single kb, but compression was not an option. I'm pretty sure the answer is never. It was never in the 90s, it is extra never now that we have hard drives hundreds of gb big.
UTF-8 is better because bytes are a natural unit. You dont have to worry about surrogates. You dont have to worry about de-synchronization issues. You dont have to worry about byte order.
Backwards compatibility with ascii and basically every interface ever, also helps. (Well not 7bit smtp..). The fact is, ascii is everywhere. Being able to treat it as just a subset of utf-8 makes a lot of things easier.
> (eg, most emoji) require four bytes each in both systems.
This is misleading because most emoiji are not just a single astral character but a combination of many.