Comment by hnlmorg
17 hours ago
It might seem unsound but it’s the concept we’ve built our terminals around.
And there are already definitions and APIs to decide the terminal cell width of different Unicode characters.
I’m not saying it’s easy, because you end up with 3 different string widths:
1. memory length in bytes
2. Number of Unicode characters (which is annoyingly also different to the number of Unicode code points since some codes can be combined)
3. terminal cell width
But as annoying as it is, there are a few relatively mature libraries you can use to solve this problem.
> And there are already definitions and APIs to decide the terminal cell width of different Unicode characters.
And how! I know of four different and mutually-incompatible definitions/behaviours, and I’m confident there will be more; and once you get to things like Indic scripts and Right-to-Left there are no good answers, and no cell-based terminal produces even vaguely acceptable results.
As for meanings of string width, I’d count at least four: Unicode code units, Unicode scalar values, Unicode extended grapheme clusters, and terminal cells.
> And how! I know of four different and mutually-incompatible definitions/behaviours, and I’m confident there will be more;
I’ve not found that to be the case. Usually the issue is more around different libraries being out of date (ie pinned to an earlier version of Unicode)
> and once you get to things like Indic scripts and Right-to-Left there are no good answers, and no cell-based terminal produces even vaguely acceptable results.
That’s because right-to-left isn’t supported and in my opinion shouldn’t be supported my the terminal emulator itself.
Right-to-left support would be handled by the client application, not the terminal emulator. The terminal is just there to offer a grid of Unicode cells. It’s not a writing composition suite.
Just like how the terminal emulator isn’t readline, it isn’t emacs, it isn’t hexdump, and it isn’t vim.
> As for meanings of string width, I’d count at least four: Unicode code units, Unicode scalar values, Unicode extended grapheme clusters, and terminal cells.
I’d already touched on the fact that Unicode widths can vary depending on what you’re measuring. But your Unicode library would/should handle most of the pain there.
When writing software for a terminal, you’re really only interested in the characters you need to loop through, their cell width, and, in some languages, the number of bytes.
As I’ve said previously, I’ve written terminal emulators targeting a variety of different renderers like SDL (so not just js+canvas), as well as complex terminal software like $SHELLs. From what I’ve built over the decades, honestly Unicode support isnt as big problem as it used to be. Sure, there’s plenty of footguns that you need to be aware of, but that’s just the nature of working with technology originally designed in the 60s/70s. Like how you need to be aware of cell widths if you intend to do any manual text alignment in client software. But at least there are libraries to make that task easier.
And if you don’t case about text alignment, then the terminal emulator will handle that for you.
There are several quite different things called wcwidth, in part because as a POSIX function it wasn’t well-specified; now I think there are two main families of it, maybe still three; https://wcwidth.readthedocs.io/en/latest/specs.html is the best-specified and -considered of them, but not the most popular. Then there are others that see how wide a character renders, and quantise in some fashion to cell. (This obviously requires knowledge of the font, which should be unavailable to the shell, so it’s a technique only terminals can use.) And in all of these, there are multiple incompatible versions hanging around. The aforelinked one shows some of its changes over time as they wrestle with trying to get things as reasonable as possible, which is fairly obviously hopeless anyway. (Some characters are just too wide, some extremely so, some less so. U+FDFD is generally made 1×1, but it should truly be something like 2×2 or 1×8 or a little wider depending on whether the font employs https://en.wikipedia.org/wiki/Word_heaping. Or the widest Telugu consonant, y, which wants to be 2 columns by itself or 3 or 4 columns with some conjuncts and vowels (e.g. ykuu, య్కూ), but gets squished to one by itself or two in those wider cases.)
The terminal emulator must handle a lot of RtL stuff to get it acceptable. In raw mode (where dimensions are knowable) you could almost leave it to the app, but outside raw mode it’s pretty obvious that the terminal has to be BiDi-aware. But even in raw mode, you really can’t treat it as you describe, because Arabic text rendering is rather complex, full of necessary ligatures.
3 replies →