Hacker News Reader

Comment by hgs3 | Hacker News Reader

It gets worse. Windows doesn't use UTF-16 for file paths it uses UCS-2. Example: Windows paths can contain unpaired UTF-16 surrogates which is illegal in UTF-16. Unix paths are no better; they are "bags of bytes" and not necessarily UTF-8. File paths are not guaranteed to be Unicode normalized. Even if they are canonically equivalent (equal graphemes) it doesn't mean they are binary equivalent (equal code points). You need to treat file paths as their own special thing and not strings.

View on HN

Windows uses potentially-ill-formed UTF-16.

No language or environment that uses UTF-16 validates it, ever. Not a single one that I know of. But that doesn’t make it UCS-2, it just makes it potentially-ill-formed UTF-16.

I just have the impression that they s/UCS-2/UTF-16/g without changing anything. Was that not the case? Genuinely curious whether there were UTF-16-specific changes made before they declared that Windows uses/supports UTF-16.

Windows validates UTF-16 when ansi functions are called, they convert string arguments to UTF-16 and call unicode function, then convert results from UTF-16 to ansi.

That’s nothing to do with validating UTF-16. What you’re describing is converting from code pages to UTF-16, which incidentally should produce well-formed UTF-16.

And Linux paths can contain anything that isn't a nul character or a /.

Your points about normalization only matter in so far as Windows is generally setup to be case insensitive (which can be disabled).

All UTF strings face normalization issues for a whole host of reasons.

> (which can be disabled)

Wouldn't that break spectacularly? The whole "the fs is case insensitive" assumption has kind of been a given on Windows for about 40 years now.

I remember accidentally creating an NTFS flash drive with two names with difference cases in Linux. Windows programs sure didn't like that.

It was added in 2018 as a folder attribute. There isn't a global option but you can annotate something as case sensitive.

Wrong. Windows uses UTF-16. It used to use UCS-2 a long time ago.

Unpaired surrogates are not allowed in UTF-16, so whatever Windows uses is not quite it.

Windows doesn't bother checking every string after every modification, but neither does Linux with UTF-8. You can pass invalid UTF-8 to tons of APIs and almost all of them will just work, just like you can with UTF-16 in Windows.

Doing a string validation check in every single API call would waste cycles for no good reason.

Linux (the kernel) doesn't claim (AFAIK) to use UTF-8. It takes NUL-terminated strings in any charset you choose (or none) and either compares them with other NUL-terminated strings, or spits them back out again later.

Interpreting a string as encoding text in particular character set is, as far as the kernel is concerned, a problem for userspace.

Windows does make claims of "supporting" UTF-16.

Linux supports UTF-8 the same way Windows does: by trusting you that the encoding is correct, unless you access APIs that are encoding dependent somehow. The Windows API is a lot more complete than the Linux API, so comparing them is rather pointless.

Windows supports UTF-16, it doesn't guarantee UTF-16 correctness. The native methods annotated with the W suffix all take UTF-16, so unless you want to render your own fonts, you're going to need to provide it with either that or ASCII.

Reply on news.ycombinator.com