Hacker News Reader

Comment by tialaramex | Hacker News Reader

The needs to convert character encodings should be gradually going away. There's not going to be more call for a standard library to address this in 2026 than there is today.

It's more crazy that C++ was so slow to mandate UTF-8 support. You have the situation where your modern C++ environment might have several distinct "string" types none of which is guaranteed to just be UTF-8.

View on HN

The need to convert UTF-8 <-> UTF-16 is not going to go away anytime soon. Many existing formats or APIs specify one or the other.

UTF-16 is only a pain when working with Windows APIs though. Most other 'legacy APIs' I've seen so far provide easy ways to get UTF-8 encoded data in and out, even though they use UTF-16 under the hood (like [NSString stringWithUTF8String:] on macOS).

Win32 also has MultiByteToWideChar and friends, although being a C API, it's not particularly pleasant to use. For newer HSTRING-based WinRT APIs, if you're calling them from C++, you're probably going to use C++/WinRT, which provides conversion functions that operate on std::string_view and std::wstring_view.

So, you're right, I need to amend that to "mostly needed for file formats and protocols".

There are only two types of APIs, those that support UTF-8 and broken APIs.

Users should actively file bug reports with any system or product that insists on UTF-16 or abandon use of such legacy systems if they refuse to change.

A lot of the contexts where C++ has been deployed predate the shift to UTF-8 so I think there's actually more of a reason to have the support.

Say, for Japanese texts Shift-JIS is vastly more economical than UTF-8, so a number of systems uses it. Likely this is the case for some other writing systems.

UTF-16 should work in there cases eventually though.

Many of those texts include enough ASCII characters (like HTML tags) that UTF-8 is more efficient even then.

For ASCII (except the blackslash and tilde) Shift-JIS and UTF-8 are the same, one byte. So your HTML tags work just fine in Shift-JIS.

For small Japanese-only text, Shift-JIS really will be smaller, but it's just not usually enough to care, and the price is you can't do Unicode. It occupies a similar space to 8859-1 / Windows 1252 where older software uses it but you should just transition.

Oh, I meant UTF-8 would be more efficient than UTF-16.

it seems like we are stuck with UTF-16-alikes until we manage to get rid of JavaScript, Java, and Windows

Wait, JavaScript? Under the hood, JavaScript strings have a variety of encodings, but I can't think of any API that forces you to use a specific encoding.

the length property and the charAt method for example are defined to use UTF-16. length is the length in UTF-16 code units and charAt returns a UTF-16 code unit at a certain index.

That's something that should be addressed with the javascript steering committees (if such things exist) so that the standard is updated to support UTF-8.

Reply on news.ycombinator.com