Comment by hnlmorg
6 hours ago
> And how! I know of four different and mutually-incompatible definitions/behaviours, and I’m confident there will be more;
I’ve not found that to be the case. Usually the issue is more around different libraries being out of date (ie pinned to an earlier version of Unicode)
> and once you get to things like Indic scripts and Right-to-Left there are no good answers, and no cell-based terminal produces even vaguely acceptable results.
That’s because right-to-left isn’t supported and in my opinion shouldn’t be supported my the terminal emulator itself.
Right-to-left support would be handled by the client application, not the terminal emulator. The terminal is just there to offer a grid of Unicode cells. It’s not a writing composition suite.
Just like how the terminal emulator isn’t readline, it isn’t emacs, it isn’t hexdump, and it isn’t vim.
> As for meanings of string width, I’d count at least four: Unicode code units, Unicode scalar values, Unicode extended grapheme clusters, and terminal cells.
I’d already touched on the fact that Unicode widths can vary depending on what you’re measuring. But your Unicode library would/should handle most of the pain there.
When writing software for a terminal, you’re really only interested in the characters you need to loop through, their cell width, and, in some languages, the number of bytes.
As I’ve said previously, I’ve written terminal emulators targeting a variety of different renderers like SDL (so not just js+canvas), as well as complex terminal software like $SHELLs. From what I’ve built over the decades, honestly Unicode support isnt as big problem as it used to be. Sure, there’s plenty of footguns that you need to be aware of, but that’s just the nature of working with technology originally designed in the 60s/70s. Like how you need to be aware of cell widths if you intend to do any manual text alignment in client software. But at least there are libraries to make that task easier.
And if you don’t case about text alignment, then the terminal emulator will handle that for you.
There are several quite different things called wcwidth, in part because as a POSIX function it wasn’t well-specified; now I think there are two main families of it, maybe still three; https://wcwidth.readthedocs.io/en/latest/specs.html is the best-specified and -considered of them, but not the most popular. Then there are others that see how wide a character renders, and quantise in some fashion to cell. (This obviously requires knowledge of the font, which should be unavailable to the shell, so it’s a technique only terminals can use.) And in all of these, there are multiple incompatible versions hanging around. The aforelinked one shows some of its changes over time as they wrestle with trying to get things as reasonable as possible, which is fairly obviously hopeless anyway. (Some characters are just too wide, some extremely so, some less so. U+FDFD is generally made 1×1, but it should truly be something like 2×2 or 1×8 or a little wider depending on whether the font employs https://en.wikipedia.org/wiki/Word_heaping. Or the widest Telugu consonant, y, which wants to be 2 columns by itself or 3 or 4 columns with some conjuncts and vowels (e.g. ykuu, య్కూ), but gets squished to one by itself or two in those wider cases.)
The terminal emulator must handle a lot of RtL stuff to get it acceptable. In raw mode (where dimensions are knowable) you could almost leave it to the app, but outside raw mode it’s pretty obvious that the terminal has to be BiDi-aware. But even in raw mode, you really can’t treat it as you describe, because Arabic text rendering is rather complex, full of necessary ligatures.
Unicode defines the width of characters. This is what the POSIX functions like wcwidth refer to. eg https://www.unicode.org/reports/tr11/
So you shouldn't need knowledge of the type-face to know the size of the printed character. If other libraries are printing the characters and then extracting the width from that, then they're doing it wrong and you should look to use a different library instead.
The bigger problem I've run into is not different libraries making up their own width assumptions. It's been libraries built against outdated versions of the Unicode standard.
> The terminal emulator must handle a lot of RtL stuff to get it acceptable. In raw mode (where dimensions are knowable) you could almost leave it to the app, but outside raw mode it’s pretty obvious that the terminal has to be BiDi-aware. But even in raw mode, you really can’t treat it as you describe, because Arabic text rendering is rather complex, full of necessary ligatures.
I think RTL is a distraction from the actual point that we're discussing. But it's worth noting that there are already CSI sequences for enabling RTL behaviour in terminal emulators. It's not something I've personally implemented in my own projects because I have no personal need for it and have had no feature requests from people asking for it who could test the implementation's soundness (unlike Chinese characters, which have seen a reasonable degree of popularity in the projects I've built). But I'm sure some other terminals support it just fine since it was added decades ago as part of ECMA 48.
No, UAX #11 does not define the width of characters. It’s called “East Asian Width” for a reason. As the note in the Scope section begins:
> The East_Asian_Width property is not intended for use by modern terminal emulators without appropriate tailoring on a case-by-case basis. […]
It’s about a very specific typographical custom in a specific set of languages. Terminal emulators have piggy-backed onto it because it’s fairly close to what they want and definitely better than nothing, but if they limit themselves to this definition, they do a bad job.
> you shouldn't need knowledge of the type-face to know the size of the printed character
To see the most extreme example, look through Google Fonts for U+FDFD <https://fonts.google.com/?preview.text=%EF%B7%BD> and see the radical differences in aspect ratio (and size!). In some other scripts there are multiple ways of writing things which have quite different width characteristics. In English we have different accepted and understood letterforms for a, g, v, z and probably others, but they’re all roughly the same width. But this isn’t always true in other scripts. My name is written in Telugu as క్రిస్ (kri s), but that r curve can be drawn below or to the left (both seen among <https://fonts.google.com/?preview.text=%E0%B0%95%E0%B1%8D%E0...>), and even sometimes a little to the right; and you don’t actually have to write it with a conjunct, క్ రిస్ (k ri s) means the same thing and there used to be some fonts or apps that wouldn’t render conjuncts. (Encodingwise, I inserted a ZERO WIDTH NON-JOINER after the {k, suppress vowel} and before the {r, -i} to simulate this effect. And then HN turned the ZWNJ into a space. Sigh. I should ask them once again to stop doing that.) Of these three different legitimate renderings of the same text, one works in two cells, one wants two and a half, one needs three.
1 reply →