Comment by bflesch
11 hours ago
I agree. At this point scripting languages should have native support for string subtypes with custom charsets (ascii/utf16) or just regexp-defined or even ENUM()'erated values.
IIRC historically on Windows, a string was UTF16, on unix it was ASCII; nowadays everywhere it's UTF8 without a way to specifically limit what goes into a string.
For example UTF8 opens the door to homoglyph attacks and various other things (RTL, spaces), and a program should be able to force a string to be ASCII-only so that these classes of problems are ruled out.
The NT kernel predates UTF8 by at least 3 years. It originally used UCS2, which covers the basic multilingual plane of the first unicode standard.
Homoglyph attacks and RTL work just as well in UTF-16. UTF-8 and UTF-16 are different character encodings for the same character set (Unicode).
That being said, Windows permits unpaired surrogates in its "UTF-16" strings, even though that is not actually valid UTF-16. Similarly, many Linux APIs accept arbitrary bytes, not just valid UTF-8.
Of course. I was trying to say that it would be nice to limit certain variables to an ASCII charset, or ASCII+umlaut+accents without any of the 10k other UTF8 shenanigans.