Hacker News Reader

Comment by hgs3 | Hacker News Reader

> The problem, column is counted using UTF-16 code units.

Rather than fixing the problem by using code points, they made it worse by allowing any encoding. Now, not only do you need to support UTF-16, but also UTF-8 and UTF-32 [1].

[1] https://microsoft.github.io/language-server-protocol/specifi...

View on HN

You don’t need to; UTF-16 is mandatory (ugh), but support for other encodings is negotiated (see ClientCapabilities.positionEncoding and ServerCapabilities.positionEncoding).

This seems somewhat pointless, because for compatibility you need to support UTF-16 anyway, so why bother implementing any of the other encodings in addition.

Because if the server and client both use UTF-8 internally, they can get better performance by agreeing to use UTF-8.

The question is if the performance gains are noticeable enough to warrant the added implementation complexity and the added integration testing needed. It seems doubtful to me, given the general protocol overhead.

Or use UTF-8 everywhere to fix the problem. Code points have their own issues (as does UTF-8, but on balance these seem like better engineering tradeoffs).

Using UTF-8 only has better engineering trade offs if you represent text as UTF-8. If you don’t (like JavaScript) then it’s just extra complexity. It’s no better than UTF-16. Only advantage it has is it’s more common among newer languages.

Code points is the only thing that makes sense for a multi-language protocol. It’s unambiguous and every Unicode client can talk in code points, even if they use some exotic encoding.

Reply on news.ycombinator.com