Comment by tosti
11 hours ago
My gripe with new languages like this is that strings are always UTF-8 and that makes it needlessly difficult to parse HTTP headers correctly. This has led to vulnerabilities in the past.
11 hours ago
My gripe with new languages like this is that strings are always UTF-8 and that makes it needlessly difficult to parse HTTP headers correctly. This has led to vulnerabilities in the past.
I disagree. UTF-8 is the right default. Pure ASCII strings are rarely needed in modern software. Unless you know that you need ASCII, your strings should support unicode.
Internet Messages have been ASCII since Internet Messages.
Most internet protocols these days have shifted to UTF-8.
I agree. At this point scripting languages should have native support for string subtypes with custom charsets (ascii/utf16) or just regexp-defined or even ENUM()'erated values.
IIRC historically on Windows, a string was UTF16, on unix it was ASCII; nowadays everywhere it's UTF8 without a way to specifically limit what goes into a string.
For example UTF8 opens the door to homoglyph attacks and various other things (RTL, spaces), and a program should be able to force a string to be ASCII-only so that these classes of problems are ruled out.
The NT kernel predates UTF8 by at least 3 years. It originally used UCS2, which covers the basic multilingual plane of the first unicode standard.
Homoglyph attacks and RTL work just as well in UTF-16. UTF-8 and UTF-16 are different character encodings for the same character set (Unicode).
That being said, Windows permits unpaired surrogates in its "UTF-16" strings, even though that is not actually valid UTF-16. Similarly, many Linux APIs accept arbitrary bytes, not just valid UTF-8.
Of course. I was trying to say that it would be nice to limit certain variables to an ASCII charset, or ASCII+umlaut+accents without any of the 10k other UTF8 shenanigans.