The author Paul McCann (polm) is one of my favourite programmers out there!
He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁 新聞” to find some japanese sources about this.
Ghost characters reveal something about the joint semantic and phonetic nature of most Chinese characters. I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin). And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine!
It is true that most hanzi are phonosemantic compounds; however, Japanese-created kanji are mostly semantic compounds. You can still guess the meaning, but good luck trying to guess the pronunciation.
The article mentions "an example of 彁 mistakenly used in a digitized Taisho newspaper due to a faded printing of 彊", but to me that implies the symbol already existed before then.
Isn't that backwards? The nonexistent character is in the -digitized- version so presumably OCR or something got 彊 wrong, that's not saying that 彁 was used in the -print- version.
Indeed, the source link is about exactly this: a crappy scan appears to have 彁 but a better copy reveals it was 彊.
Before that, there was the case of ÿ and Ÿ in the base IBM character set (see https://en.wikipedia.org/wiki/Ÿ). The former exists in a handful of proper nouns in French and Hungarian (as in about five proper nouns), but only as a y with a diæresis, which would normally be handled with a Unicode modifier symbol. It is also used in a couple of endangered languages in South America, which would not have influenced the character getting in to a PC character set. It is thought that possibly it was supposed to represent the Dutch "ij" character, but apparently that is never written as "ÿ" in print. Ÿ seems to not be used by any language.
á (á) is also an 'a' with a diacritical acute accent. If you mean that ÿ should not have a precomposition in Unicode, well, why not, especially if it really is used in "a handful of proper nouns in French and Hungarian".
Remember, the reason we have combining marks is that that is in fact how many of these characters were composed in actual use, especially with typewriters. Heck, 1966 US-ASCII / ECMA-6 (1985), section 5, describes the use of backspace/overstrike in US-ASCII for composition of such characters! That comes from typewriter use. And that's where compose-key sequences generally come from, too.
So it's not at all surprising that given that ÿ has _some_ use, therefore a) it is a valid glyph to construct with combining diaeresis, and b) that it has a precomposed codepoint in Unicode.
I've seen Dutch comics for kids which use ÿ instead of ij. E.g.: "mÿn". In typical comic font (which is closer to handwriting), it looks similar to ij, but it's clearly ÿ.
Surely those aforementioned non-initial cases would sometimes find themselves in a piece of all-uppercase text? You'd have found such things in print media even before the advent of computers.
Not everything is upper/lower convertible like ß for example (as far as I know)
At least until 2017, I imagine there are others though.
> In 2017, the Council for German Orthography officially adopted a capital form ⟨ẞ⟩ as an acceptable variant, ending a long debate. https://en.wikipedia.org/wiki/%C3%9F
iOS at least renders it as an n with dieresis; is that how it was intended (I’m unfamiliar with musical notation)? If so, what is so difficult about it?
In fact, (semantics aside, from a technical perspective) the preference should always be for modifiers rather than standalone characters because the chances of being supported by the viewer’s font are much greater: it doesn’t need a separate glyph explicitly drawn and add to the font file for the code point. Difficulties in entering it or typing it out should be mitigated with client-side affordances in the UI, shortcuts, etc.
n̈ renders fine on my browser, but there is no compose-key sequence for it on my system.
I suspect the reason it renders fine is that 'n' in the font I'm using is small enough that the combining mark can be rendered by "overstriking" a diaeresis.
It occurs to me that we can use 彊 to mean "a completely unknown concept that cannot be named". For example if you ask, "when Cthulhu rises from its slumber, what is thefirst thing it will do? Probably it will 彊.
Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article...
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
> apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism
Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
Yes. As far as I understood it, the result was an incoherent mess.
I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now.
And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work.
Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
Meanwhile, Latin a and Cyrillic а, which look identical, have the same origin and are often pronounced identically too, get separate codepoints. CJK unification was not based on any clear preestablished principles but simply an attempt to cling to 16-bit codepoints, Asian languages be damned.
> The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
Note that Han Unification has a history predating Unicode, and the Unicode work very much followed on from that.
The most notable is CCCII, developed in Taiwan in the early 80s, and then standardised in various places.
That's not to say that it hasn't been controversial (it has!), nor to say that it hasn't caused problems (it has!), but it's also unfair to say that it's just Unicode doing its own special thing.
It is a blunder that Japan could not see that they just copied other characters in to JIS and it’s also why we see duplicated Romaji characters within Unicode. The duplication is frankly bad regardless of number of bits need to encode although with 16 it was indeed a requirement.
See also https://en.wikipedia.org/wiki/A_Book_from_the_Sky (a 1988 art project based on intentionally making up ghost characters by hand and printing a whole book with nonexistent but aesthetically plausible Chinese characters). Amusingly, Wikipedia mentions that over a dozen of them (out of about 4,000) were later found to accidentally exist!
There's some fan fiction that treats the characters as names of ghosts and comes up with back stories and visualizations for them (the article links to a compilation, but it's 403ing for me). Nothing mainstream though.
It’s really a complete and total nothing burger. Extra code points were added, might have been an issue when we were trying to cap the total number below needing some arbitrarily fixed number of bytes for convenience, but now that’s no longer the case and they’re just a historical oddity that costs nothing to maintain and certainly don’t “haunt” in the sense of “ keep popping up and causing problems” in any way.
I appreciate the article nevertheless, of course, but I do feel that it would probably be more meaningful to someone that has at least a basic understanding of Japanese.
> Why didn't they simly replace the original bad one?
Starting with version 2.1 they made it so that code points can't ever be unassigned. People were getting really upset with the removals, since they can result in old data being misinterpreted. (Er, nothing actually was removed in 2.1, but it probably didn't become official policy until 3.0… ?)
Besides, it's useful to have the bad characters in there for people who want to write posts like this one discussing them.
But it's even more useful for people who want to use the actual character to be able to use it! And you can discuss using other means, not like drawings or old standard data disappears
Weird absolutism re error preservation instead of striving for correctness
> Not that hard to imagine, OCR existed back then?
How do you train OCR on a character that doesn't exist though? Even these days, OCR often confuses the 52 Latin characters, so I wouldn't expect as good of results with older technology and some 20k CJK characters.
I don't know if this is how OCR would work here, but they're made of components called radicals. It's kind of like how letters make words, but in a two dimensional layout, and one letter was wrong and a nonexistent word was made.
For example 彁 is made of parts of 引 and 歌 (the first that popped into my mind).
IIRC there's only like 200-something radicals, though some can be hard to spot due to how they overlap or nest inside each other.
"slow" - wasn't like they were pressed for time. It took them almost 20years to even start the investigation!
Also not certain you needed that much reliability compared to what was available to match one pic to a quality scan to get a reasonable number of candidates
The author Paul McCann (polm) is one of my favourite programmers out there!
He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
[0] https://github.com/polm/fugashi [1] https://www.japanesenlp.com/ [2] https://spacy.io/
(2018)
Major previous discussions:
110 comments: https://news.ycombinator.com/item?id=32095502
I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁 新聞” to find some japanese sources about this.
Ghost characters reveal something about the joint semantic and phonetic nature of most Chinese characters. I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin). And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine!
It is true that most hanzi are phonosemantic compounds; however, Japanese-created kanji are mostly semantic compounds. You can still guess the meaning, but good luck trying to guess the pronunciation.
https://en.wikipedia.org/wiki/Kokuji
> I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin).
Huh? How do you pronounce 切?
> And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine!
Why is "elder brother" a meaning hint if you've already assumed that 哥 is the phonophore?
3 replies →
The article mentions "an example of 彁 mistakenly used in a digitized Taisho newspaper due to a faded printing of 彊", but to me that implies the symbol already existed before then.
Isn't that backwards? The nonexistent character is in the -digitized- version so presumably OCR or something got 彊 wrong, that's not saying that 彁 was used in the -print- version.
Indeed, the source link is about exactly this: a crappy scan appears to have 彁 but a better copy reveals it was 彊.
3 replies →
Before that, there was the case of ÿ and Ÿ in the base IBM character set (see https://en.wikipedia.org/wiki/Ÿ). The former exists in a handful of proper nouns in French and Hungarian (as in about five proper nouns), but only as a y with a diæresis, which would normally be handled with a Unicode modifier symbol. It is also used in a couple of endangered languages in South America, which would not have influenced the character getting in to a PC character set. It is thought that possibly it was supposed to represent the Dutch "ij" character, but apparently that is never written as "ÿ" in print. Ÿ seems to not be used by any language.
> but only as a y with a diæresis
á (á) is also an 'a' with a diacritical acute accent. If you mean that ÿ should not have a precomposition in Unicode, well, why not, especially if it really is used in "a handful of proper nouns in French and Hungarian".
Remember, the reason we have combining marks is that that is in fact how many of these characters were composed in actual use, especially with typewriters. Heck, 1966 US-ASCII / ECMA-6 (1985), section 5, describes the use of backspace/overstrike in US-ASCII for composition of such characters! That comes from typewriter use. And that's where compose-key sequences generally come from, too.
So it's not at all surprising that given that ÿ has _some_ use, therefore a) it is a valid glyph to construct with combining diaeresis, and b) that it has a precomposed codepoint in Unicode.
I've seen Dutch comics for kids which use ÿ instead of ij. E.g.: "mÿn". In typical comic font (which is closer to handwriting), it looks similar to ij, but it's clearly ÿ.
> Ÿ seems to not be used by any language.
Surely those aforementioned non-initial cases would sometimes find themselves in a piece of all-uppercase text? You'd have found such things in print media even before the advent of computers.
Not everything is upper/lower convertible like ß for example (as far as I know)
At least until 2017, I imagine there are others though.
> In 2017, the Council for German Orthography officially adopted a capital form ⟨ẞ⟩ as an acceptable variant, ending a long debate. https://en.wikipedia.org/wiki/%C3%9F
Sure, but I've never heard of a diæresis being added to a capital letter.
2 replies →
Great, so they added Ÿ but not n̈, making it very difficult to write music reviews.
(Not sure how the latter will render in your browser).
iOS at least renders it as an n with dieresis; is that how it was intended (I’m unfamiliar with musical notation)? If so, what is so difficult about it?
In fact, (semantics aside, from a technical perspective) the preference should always be for modifiers rather than standalone characters because the chances of being supported by the viewer’s font are much greater: it doesn’t need a separate glyph explicitly drawn and add to the font file for the code point. Difficulties in entering it or typing it out should be mitigated with client-side affordances in the UI, shortcuts, etc.
3 replies →
n̈ renders fine on my browser, but there is no compose-key sequence for it on my system.
I suspect the reason it renders fine is that 'n' in the font I'm using is small enough that the combining mark can be rendered by "overstriking" a diaeresis.
It occurs to me that we can use 彊 to mean "a completely unknown concept that cannot be named". For example if you ask, "when Cthulhu rises from its slumber, what is thefirst thing it will do? Probably it will 彊.
That's not on my keyboard... Can I just stick with "ineffable"?
Not exactly, it doesn't look like the definition or even connotations of "ineffable" include "unknown".
7 replies →
More mundanely, you could also use it as a marker character, given that nobody except for sneaky little QA hobbitses will ever use it in their input.
So www.catb.org/~esr/jargon/html/M/mu.html
This sounds like a SCP entry
Perhaps the meaning of this character is sealed in some vault
This would make a great tattoo, as a pun on people getting Kanji character tattoos without being able to read Kanji.
Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article...
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
> apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism
Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
Yes. As far as I understood it, the result was an incoherent mess.
I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now.
And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work.
Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
27 replies →
Meanwhile, Latin a and Cyrillic а, which look identical, have the same origin and are often pronounced identically too, get separate codepoints. CJK unification was not based on any clear preestablished principles but simply an attempt to cling to 16-bit codepoints, Asian languages be damned.
6 replies →
There is a nice list of example Chinese characters that differ between languages, e.g., Chinese vs Korean vs Japanese, etc: https://en.wikipedia.org/wiki/Han_unification#Examples_of_la...
And then some examples of non-unified Chinese chars: https://en.wikipedia.org/wiki/Han_unification#Examples_of_so...
> The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
Note that Han Unification has a history predating Unicode, and the Unicode work very much followed on from that.
The most notable is CCCII, developed in Taiwan in the early 80s, and then standardised in various places.
That's not to say that it hasn't been controversial (it has!), nor to say that it hasn't caused problems (it has!), but it's also unfair to say that it's just Unicode doing its own special thing.
What is the rationale to the unifying direction of changes in Unicode?
The Han unification probably was about fitting CJK into the 16 bits aka BMP. It was a blunder indeed.
It is a blunder that Japan could not see that they just copied other characters in to JIS and it’s also why we see duplicated Romaji characters within Unicode. The duplication is frankly bad regardless of number of bits need to encode although with 16 it was indeed a requirement.
Fascinating. But, I guess it's better to have superfluous invalid characters than missing real ones.
Xu Bing has a book that consists entirely of invented characters:
https://en.wikipedia.org/wiki/A_Book_from_the_Sky
(2018)
I've seen YouTube videos on this topic before.
> The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me.
It's from Unicode version 3.1 (published 2001) so this is surprising.
See also https://en.wikipedia.org/wiki/A_Book_from_the_Sky (a 1988 art project based on intentionally making up ghost characters by hand and printing a whole book with nonexistent but aesthetically plausible Chinese characters). Amusingly, Wikipedia mentions that over a dozen of them (out of about 4,000) were later found to accidentally exist!
Is anyone using these characters now for anything? No youth language or online slang using it?
There's some fan fiction that treats the characters as names of ghosts and comes up with back stories and visualizations for them (the article links to a compilation, but it's 403ing for me). Nothing mainstream though.
It's something! I propose that the first 4 of those characters should in English be called Blinky, Pinky, Inky, and Clyde.
Should probably have "(2008)" in the title
Saw headline, expected branch prediction vulnerability involving Unicode, was surprised at a completely different topic.
A language where you can make up new characters seems anathema to a universal character encoding.
It’s really a complete and total nothing burger. Extra code points were added, might have been an issue when we were trying to cap the total number below needing some arbitrarily fixed number of bytes for convenience, but now that’s no longer the case and they’re just a historical oddity that costs nothing to maintain and certainly don’t “haunt” in the sense of “ keep popping up and causing problems” in any way.
I appreciate the article nevertheless, of course, but I do feel that it would probably be more meaningful to someone that has at least a basic understanding of Japanese.
So what is the spectre? I thought this was proposing something new like the manifesto
> The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me
Why didn't they simly replace the original bad one?
> nine hundred pages. Imagine tracking down a single character without a page reference
Not that hard to imagine, OCR existed back then?
> Why didn't they simly replace the original bad one?
Starting with version 2.1 they made it so that code points can't ever be unassigned. People were getting really upset with the removals, since they can result in old data being misinterpreted. (Er, nothing actually was removed in 2.1, but it probably didn't become official policy until 3.0… ?)
Besides, it's useful to have the bad characters in there for people who want to write posts like this one discussing them.
But it's even more useful for people who want to use the actual character to be able to use it! And you can discuss using other means, not like drawings or old standard data disappears Weird absolutism re error preservation instead of striving for correctness
2 replies →
> Not that hard to imagine, OCR existed back then?
How do you train OCR on a character that doesn't exist though? Even these days, OCR often confuses the 52 Latin characters, so I wouldn't expect as good of results with older technology and some 20k CJK characters.
(Talking about Japanese here)
I don't know if this is how OCR would work here, but they're made of components called radicals. It's kind of like how letters make words, but in a two dimensional layout, and one letter was wrong and a nonexistent word was made.
For example 彁 is made of parts of 引 and 歌 (the first that popped into my mind).
IIRC there's only like 200-something radicals, though some can be hard to spot due to how they overlap or nest inside each other.
1 reply →
It does exist? It's part of Unicode!
2 replies →
OCR was slow and unreliable and was for a very long time.
"slow" - wasn't like they were pressed for time. It took them almost 20years to even start the investigation! Also not certain you needed that much reliability compared to what was available to match one pic to a quality scan to get a reasonable number of candidates
"- the spectre of communism. All the powers of old encoding have entered into a holy alliance to exorcise this spectre..."
Well, Communism made it into Unicode (U+262D). I tried to show it here but it gets stripped.
I have it it in my .XCompose as
Never used, but I laugh every time I see it there.
As a slightly related tangent, The compose key mnemonic interface for rarely used characters is pretty great, beats trying to remember alt codes.
1 reply →
Figures, it's a capitalist website after all...
1 reply →
[flagged]
[flagged]
[flagged]
[flagged]