Hacker News Reader

Comment by bitwize | Hacker News Reader

Before "developers, developers, developers" comes "backward compatibility, backward compatibility, backward compatibility". Windows is perhaps the first commercial platform to commit to Unicode; they made that commitment when UTF-8 was still some notes scribbled on Brian Kernighan's napkin. And all future Win32 implementations must be 100% binary compatible with previous ones. That creates inertia for UTF-16 (or UCS-2), true, but the backwards compatibility guarantees make Windows an absolute joy compared to Linux if you want to write software with a long service lifetime. The decision to stick with 16-bit Unicode is an engineering tradeoff.

View on HN

but the backwards compatibility guarantees make Windows an absolute joy compared to Linux if you want to write software with a long service lifetime

The linux user-space ABI is extremely stable. I think the one thing that would frustrate the development of long service lifetime software on linux would just be library availability on various distros, but the actual operating environment presented by the kernel and the image loader for user-space software on linux is very, very stable.

And all future Win32 implementations must be 100% binary compatible with previous ones

And no one is asking microsoft to break the windows user-space ABI. Adding a new, sufficiently-tested, MBCS codepage would have no impact at all on existing windows software. None whatsoever. Other than to make localization a whole lot easier. And compared to the cost of building in a whole new csrss-level subsystem like the one Microsoft just built in to windows last month (linux), it's probably a lot safer and easier to test.

Working in this world myself, windows developers become philosophical when discussing localization. "Someday, we'll turn on UNICODE," "We really should be using TCHAR" (as if that silly thing would fix anything at all), "Shouldn't we really be using wstring?"

OSX, Linux, and mobile developers just do it. It's mostly a solved problem on those platforms.

The decision to stick with 16-bit Unicode is an engineering tradeoff.

It's a cost tradeoff - Microsoft doesn't want to spend the developer and testing time adding a UTF-8 codepage.

Vote here: https://wpdev.uservoice.com/forums/266908-command-prompt-con....

Currently we do "support" CP65001 in the console, but things break if you enable it. One of the problems, for example, is that .NET sees 65001 and starts outputting the UTF-8 BOM everywhere, breaking applications that don't even care about the character encoding. I suspect that's going to be difficult to fix without breaking compatibility.

Having said that, I think it's apparent that we are investing heavily in the console for the first time in a long while, so I'm more hopeful than ever that we can get this fixed.

Well, if I could use UTF-8 with CreateFileA() et. al. and never have to use wchar_t again, that would just be like Christmas. I don't know whether this survey is for blanket win32 UTF-8 compatibility or whether this just means making the admittedly hugely improved cmd.exe work with UTF-8 command parsing, but anything that gives us better UTF-8 support is a step in the right direction.

The BOM is an aBOMination. If you simply assume that everything is UTF-8 until proven otherwise, you can get pretty far - the legacy code pages produce text that's not usually valid UTF-8 unless they stick to ASCII.

Yes, you can assume, but existing applications, maybe written in 1990s even 1980s, won't. And there are millions of computers, maybe in important industrial companies, are still using them.

If that's the case they won't handle a BOM either.

the actual operating environment presented by the kernel and the image loader for user-space software on linux is very, very stable.

The guarantees that Windows makes about backwards-compatibility go far beyond just the image-loader and syscalls. For comparison, the glibc backwards-compatibility story is... messier. https://www.kernel.org/pub/software/libs/glibc/hjl/compat/

Linux had the luxury of waiting until UTF-8 was available before settling on a Unicode strategy. Given that UTF-8 was extremely compatible both forwards and backwards, it was a fortuitous choice. I weep a little whenever I think about how Microsoft messed up by being an early adopter of Unicode - being an early adopter should have made life better, not worse.

True story: Office maintained its API to be compatible with a plugin which its developer has already bankrupted, and its source code is lost. Because millions of US Government computers are still using them.

Reply on news.ycombinator.com