Comment by hmry
21 hours ago
In the eternal words of the Microsoft Game Development Kit docs[0], "Windows operates natively in UTF-16 [...]. This is a unique burden that Windows places on code that targets multiple platforms."
I wonder when Microsoft will stop being a "UTF-16 house" adding UTF-16 to new projects... They did it again to LSP, and that was in 2016. Maybe by 2045 Microsoft will use UTF-8 for new projects, and then by 2100 we can switch to UTF-8 by default.
[0] https://learn.microsoft.com/en-us/gaming/gdk/docs/gdk-dev/ov...
And again in 2026: https://github.com/microsoft/pylance-release/blob/main/docs/...
> They did it again to LSP, and that was in 2016.
That one is explainable by the fact that it was developed for VS Code, a typescript/javascript project. Javascript uses UTF-16 for strings (for historical reasons unrelated to Microsoft, though they are the same reasons that made MS use UCS-2 during the same time period).
> Javascript uses UTF-16 for strings
I don’t think the spec requires that. charAt indexes on UTF-16 code points, but I don’t remember ever seeing that it has to do that in O(1) time, so that doesn’t imply the implementation has to use UTF-16 internally.
Also, for modern JavaScript, https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe... says:
“Note: Avoid re-implementing the solutions above using charAt(). The detection of lone surrogates and their pairing is complex, and built-in APIs may be more performant as they directly use the internal representation of the string. Install a polyfill for the APIs mentioned above if necessary.”
Because of that, I would hope most modern implementations to at least have a variant internal representation that is similar to UTF-8.
https://en.wikipedia.org/wiki/UTF-16
> UTF-16 is used by the Windows API, and by many programming environments such as Java and Qt. The variable-length character of UTF-16, combined with the fact that most characters are not variable-length (so variable length is rarely tested), has led to many bugs in software, including in Windows itself.
Absolutely, UTF-16 is bad and inexcusable in most contexts. Perhaps you misinterpreted my comment or I wasn't expressing myself properly.
I was trying to say: If all the components in your stack are javascript-based, it makes sense that your RPC-API would also use UTF-16. It wasn't Microsoft pushing it for historical Windows reasons as GP implied, it was dictated by a JS stack they chose but didn't create.
2 replies →