Comment by cyphar
4 years ago
JIS, Big5, UHC, and GB all use a codepoint-to-character approach. You're right to point out that many aspects of "multilingual" support are written by people who do not know another language and so end up being hopelessly misguided but it's not really fair to say that Unicode invented this and thrust it upon the CJK world. Every pre-existing system of representing 漢字 had a codepoint table (which Unicode references in their description of each character).
Han Unification was in my view problematic but was driven by technical limitations (then again, if Simplified Chinese characters had also been unified I suspect there would've been more pushback to come up with a better solution, but ultimately Japanese was stuck with being the only one making a major compromise on that front).
I don't think a stroke based or combination system would've been better for many reasons: https://news.ycombinator.com/item?id=32102093. And if you don't trust Americans who at least tried to learn about the subject matter, how much do you trust any other programmer (who has no interest in other languages) to be able to handle a more complicated system for representing and rendering 漢字?
I don't know about Chinese since I'm barely literate in it.
But copying how every Chinese dictionary renders characters as trees of simpler characters seems like a much better approach than the arbitrary letter to number mapping of unicode. That it works well in English is only due to the fact it has no accents.
I can confidently say that native solutions for scripts with accents were _not_ unicode like but overstrike. My grandfather has the source code, in Romanian, of a 1960s computer payment system he worked on which had to deal with both Romanian and Hungarian names.
The combinatorial explosion of possible letters and accents made unicode like encodings an obvious non-starter. Historically names could pick up any accent (some times more than one) on any letter. When you have 26 base letters and 6 possible accents you'd need 26 + 26 * 6 (182) unique representations for single accented letters and 26 + 26 * 6 + 26 ^ 2 (1118) for double accented letters. That makes a language which is fundamentally alphabet based unusable on a keyboard. Something that Unicode is still sweeping under the rug.
> But copying how every Chinese dictionary renders characters as trees of simpler characters seems like a much better approach than the arbitrary letter to number mapping of unicode.
Then why does every natively-developed encoding system in a 漢字-using country not do it that way? For one thing, how would you handle the fact that 食反＝飯 but 食耳＝餌 (and the correct rendering depends on the language and locale of the text being rendered)? How about 辵 (aka 辶, ⻍, ⻌)? There are many issues on top of this one, but this is among the most obvious. In the end you would end up with having your encoding format look like Ideographic Description Sequences (which exist in Unicode) but every rendering library would need to have its own lookup table anyway to produce the correct character. Overlaying accents on top of latin characters (in most European languages) is nowhere near as complicated as combining components to form 漢字.
> I can confidently say that native solutions for scripts with accents were _not_ unicode but overstrike.
Unicode supports combining characters for this reason, though there are separate problems with this approach (some characters look almost identical but semantically should be treated differently -- maybe that is something fonts could deal with, but I suspect "Latin Unification" would've gotten more pushback than Han Unification did). If we want computer systems from different languages and cultures to interoperate there are going to be a few rough edges.
> When you have 26 base letters and 6 possible accents you'd need 26 + 26 * 6 (182) unique representations for single accented letters and 26 + 26 * 6 + 26 ^ 2 (1118) for double accented letters.
No, you don't. Only the most common combinations have their own Unicode number. Most combinations can simply be combined by base and accent ("Mark") numbers. Unicode is not that stupid.
>No, you don't. Only the most common combinations have their own Unicode number. Most combinations can simply be combined by base and accent ("Mark") numbers. Unicode is not that stupid.
https://en.wikipedia.org/wiki/List_of_Unicode_characters#Lat...
The most common being literally all of them.
Between Latin-1 Supplement, Latin Extended-A, Latin Extended-B and Latin Extended Additional you have some 700 extra characters of which half are some type of accented letter. I only said you'd need 182 for the six most common European accents. Unicode somehow ends up using 300.
The only people who defend unicode are people who have never looked into the spec.
> Every pre-existing system of representing 漢字 had a codepoint table (which Unicode references in their description of each character).
Even typewriters worked with a giant table: https://en.wikipedia.org/wiki/Chinese_typewriter