Comment by jcgl
1 day ago
I think you're mistaking text-with-structured data for structured data itself.
Because unix shell is irrevocably text-oriented, kludging in something like JSON is basically the best that can be done when you start to want to do structured operations on structured data. (I'm sympathetic to your point about the AWS CLI tools doing JSON by default though--that just sounds like bad design.)
Being text-oriented imposes drastic limits on composability. Because there is no structure, every element of a pipeline needs to do its own parsing of the input data. This leads to brittle pipelines where every element is tightly coupled to its input's textual representation.
As an exercise, try to write a pipeline that sorts podman images by size without removing the column headers[0]:
$ podman image ls --all
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/prom/prometheus latest 937690d77350 2 months ago 367 MB
quay.io/keycloak/keycloak latest da9433c9fac3 2 months ago 466 MB
registry.fedoraproject.org/fedora-toolbox 43 a32da54355ca 4 months ago 2.19 GB
docker.io/powerdns/pdns-auth-49 latest 8c1385c9deed 4 months ago 208 MB
docker.io/testcontainers/ryuk 0.13.0 b75bc7ce94c3 6 months ago 7.21 MB
As far as I can tell, there is no way to do this in a manner that's even remotely composable. Your best bet is to basically do everything from within awk. Whatever the result would be, it certainly won't be pretty!
Contrast that with what you can do in PowerShell. You can write a couple of standalone functions[0] that are readable and composable, resulting in this pipeline:
podman image ls --all |
Replace-SpacesWithTabs |
ConvertFrom-Csv -Delimiter "`t" |
Sort-Object -Property {Convert-HumanSizeToBytes -Size $_.size} -Descending
[0] Repurposing this from a blog post I wrote: https://www.cgl.sh/blog/posts/sh.html#this-should-be-basic
the point is well-taken, but i do want to show the bash version just for fun:
this is _still_ all text, and we're relying heavily on sort to do a bunch of internal parsing and be in agreement with podman about how sizes should be formatted. also, for "real world" work, i dunno if the tee trick here has any kind of order guarantees, just that it works fine in this case. I'd probably just end up dropping the header and living with worse output in reality
Ooh, I was so hoping someone would take up the challenge! This is a far shorter answer than I had honestly thought was possible. More readable too, somehow? Great use of tee that I would never have come up with (though I hear what you say about there maybe not being ordering guarantees).
Unfortunately, it's not 100% correct, due to misaligned headers:
I think that speaks to your final point, which is spot-on:
> I'd probably just end up dropping the header and living with worse output in reality
This pretty much sums up plain text and unix shell imo. It's very much the pragmatic solution here, and it's what ~100% of shell scripters would choose to do. And it should make anyone question the orthodoxy around the "power" of plain text in shells.
A variant:
I used docker since that's what I have installed and I assume the output is equivalent.
sort's -t is set to tab for field separation.
stdbuf sets sed's output to only buffer a line at a time and flush, so the head in the {...} command group doesn't completely consume stdin's contents before it's passed to tail.
The column command recreates the space-aligned table based on tab-delimited input.
There is some cool stuff here.
I like using column to format the table. Appending it to alloyed's command fixes their header problem.
The stdbuf to multi-command block (term.?) is a neat trick. Although, one time when I ran this, I only got a couple lines of output. No idea why and I can't replicate it, but there could be some flakiness that results from the buffering somehow?
Question: how do the ? markers on the sort and column invocations work/what do they do?
2 replies →
…or was the point more about doing it in a pipeline?
Yes, the point was about doing it in a pipeline. The pipeline is the basis for composition of plain text in the unix shell. If something as basic as sorting a table is hard to do, it should make us question just how good the unix shell/plain text philosophy actually is.
Baking --sort flags into shell tools is a sign that the tools do not compose well.