Comment by hnfong
1 day ago
Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article...
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
> apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism
Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
Yes. As far as I understood it, the result was an incoherent mess.
I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now.
And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work.
Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
Han-unification was the result of Asian scholars, in particular the Ideaographic Research Group [1]
I have no idea what "Aristotelian essentialism" is supposed to mean, or if you are saying that the unification was that.
> And apparently the Japanese complained bitterly over the ones that were merged.
and
> but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode [...] mostly due to vocal objections from the Japanese.
Seems to contradict each other.
[1] https://en.wikipedia.org/wiki/Ideographic_Research_Group
9 replies →
> I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode
What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts?
i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints.
Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96
9 replies →
I don't know but last I glanced at it, the Unification was implied to be coming from Chinese pressure than Japanese. Each of Kanji/Hanzi countries already had each own acceptable common use character sets and those could have been just populated into each own spaces. That seemed like the original plan at some point.
There has to be a reason why it happened(as to why Unicode suddenly started insisting it has to fit inside a 2^16 total chars or whatever).
Do you have any specific examples? I never heard this before.
Interesting. I guess there are no well working normalization functions as there are for the different code points that can make umlauts (äöü)?
>having different code points so that things like text search sometimes don't work.
Cross language search seems like a hack to me. Searching in Chinese should find Chinese words and searching in Japanese should find Japanese words. Being able to search in Japanese and get Chinese results is not what most users want, unless they don't have a proper keyboard.
3 replies →
Meanwhile, Latin a and Cyrillic а, which look identical, have the same origin and are often pronounced identically too, get separate codepoints. CJK unification was not based on any clear preestablished principles but simply an attempt to cling to 16-bit codepoints, Asian languages be damned.
Cling to 16 bit codepoints sounds like a legitimate explanation? Latin and Cyrillic didn't would have saved no relevant amount of space.
It looks like the CJK unified space is over 20,000 characters, so that's a real technical magnitude distinction compared to Latin and Cyrillic. "Asian languages be damned" seems like a bad faith read, compared to "Java and Windows char is 16 bits and that will never change realistically" (and in fact they still haven't, even in 2026 things which rely on UTF16 instead of UCS2 are still commonly bugged unfortunately)
5 replies →
There is a nice list of example Chinese characters that differ between languages, e.g., Chinese vs Korean vs Japanese, etc: https://en.wikipedia.org/wiki/Han_unification#Examples_of_la...
And then some examples of non-unified Chinese chars: https://en.wikipedia.org/wiki/Han_unification#Examples_of_so...
> The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
Note that Han Unification has a history predating Unicode, and the Unicode work very much followed on from that.
The most notable is CCCII, developed in Taiwan in the early 80s, and then standardised in various places.
That's not to say that it hasn't been controversial (it has!), nor to say that it hasn't caused problems (it has!), but it's also unfair to say that it's just Unicode doing its own special thing.
What is the rationale to the unifying direction of changes in Unicode?
The Han unification probably was about fitting CJK into the 16 bits aka BMP. It was a blunder indeed.
It is a blunder that Japan could not see that they just copied other characters in to JIS and it’s also why we see duplicated Romaji characters within Unicode. The duplication is frankly bad regardless of number of bits need to encode although with 16 it was indeed a requirement.