Hacker News Reader

Comment by CountSessine | Hacker News Reader

For a company that claims to be so supportive of "developers, developers, developers", Microsoft's stubborn and developer-hostile approach to internationalization and their dogged loyalty to the awful UTF-16 encoding is ironic.

The Right Thing To Do at this point is to make UTF-8 a multi-byte code page in Windows and build a UTF-8 implementation in the msvc libc. The milquetoast excuse I hear from Microsoft people is that some win32 APIs can't handle MBCS encodings with more than 3 bytes per character. Which sort of sounds like a problem for developers to fix; perhaps Microsoft could hire some?

View on HN

Before "developers, developers, developers" comes "backward compatibility, backward compatibility, backward compatibility". Windows is perhaps the first commercial platform to commit to Unicode; they made that commitment when UTF-8 was still some notes scribbled on Brian Kernighan's napkin. And all future Win32 implementations must be 100% binary compatible with previous ones. That creates inertia for UTF-16 (or UCS-2), true, but the backwards compatibility guarantees make Windows an absolute joy compared to Linux if you want to write software with a long service lifetime. The decision to stick with 16-bit Unicode is an engineering tradeoff.

but the backwards compatibility guarantees make Windows an absolute joy compared to Linux if you want to write software with a long service lifetime

The linux user-space ABI is extremely stable. I think the one thing that would frustrate the development of long service lifetime software on linux would just be library availability on various distros, but the actual operating environment presented by the kernel and the image loader for user-space software on linux is very, very stable.

And all future Win32 implementations must be 100% binary compatible with previous ones

And no one is asking microsoft to break the windows user-space ABI. Adding a new, sufficiently-tested, MBCS codepage would have no impact at all on existing windows software. None whatsoever. Other than to make localization a whole lot easier. And compared to the cost of building in a whole new csrss-level subsystem like the one Microsoft just built in to windows last month (linux), it's probably a lot safer and easier to test.

Working in this world myself, windows developers become philosophical when discussing localization. "Someday, we'll turn on UNICODE," "We really should be using TCHAR" (as if that silly thing would fix anything at all), "Shouldn't we really be using wstring?"

OSX, Linux, and mobile developers just do it. It's mostly a solved problem on those platforms.

The decision to stick with 16-bit Unicode is an engineering tradeoff.

It's a cost tradeoff - Microsoft doesn't want to spend the developer and testing time adding a UTF-8 codepage.

Vote here: https://wpdev.uservoice.com/forums/266908-command-prompt-con....

Currently we do "support" CP65001 in the console, but things break if you enable it. One of the problems, for example, is that .NET sees 65001 and starts outputting the UTF-8 BOM everywhere, breaking applications that don't even care about the character encoding. I suspect that's going to be difficult to fix without breaking compatibility.

Having said that, I think it's apparent that we are investing heavily in the console for the first time in a long while, so I'm more hopeful than ever that we can get this fixed.

The BOM is an aBOMination. If you simply assume that everything is UTF-8 until proven otherwise, you can get pretty far - the legacy code pages produce text that's not usually valid UTF-8 unless they stick to ASCII.

Yes, you can assume, but existing applications, maybe written in 1990s even 1980s, won't. And there are millions of computers, maybe in important industrial companies, are still using them.

If that's the case they won't handle a BOM either.

Well, if I could use UTF-8 with CreateFileA() et. al. and never have to use wchar_t again, that would just be like Christmas. I don't know whether this survey is for blanket win32 UTF-8 compatibility or whether this just means making the admittedly hugely improved cmd.exe work with UTF-8 command parsing, but anything that gives us better UTF-8 support is a step in the right direction.

the actual operating environment presented by the kernel and the image loader for user-space software on linux is very, very stable.

The guarantees that Windows makes about backwards-compatibility go far beyond just the image-loader and syscalls. For comparison, the glibc backwards-compatibility story is... messier. https://www.kernel.org/pub/software/libs/glibc/hjl/compat/

Linux had the luxury of waiting until UTF-8 was available before settling on a Unicode strategy. Given that UTF-8 was extremely compatible both forwards and backwards, it was a fortuitous choice. I weep a little whenever I think about how Microsoft messed up by being an early adopter of Unicode - being an early adopter should have made life better, not worse.

True story: Office maintained its API to be compatible with a plugin which its developer has already bankrupted, and its source code is lost. Because millions of US Government computers are still using them.

Especially hilarious how SQL server nchar is widechar so most all string data is twice as large as it needs to be. But there isn't a utf8 option to enable! Though this might have been addressed in one of the later versions, I wouldn't be surprised if it was limited to enterprise edition or other crap.

Bloody hell... I looked this up as I was certain it couldn't still be the case. But yes, it is!

This was logged as an issue on Connect in 2008, [1] and Microsoft's response was:

"Thanks for your suggestion. We are considering adding support for UTF8 in the next version of SQL Server. It is not clear at this point if it will be a new type or integrate it with existing types. We understand the pain in terms of integrating with UTF8 data and we are looking at ways to effectively resolve it."

1. https://connect.microsoft.com/SQLServer/feedback/details/362...

Reply on news.ycombinator.com