The aws cli has a set of porcelain for s3 access (aws s3) and plumbing commands for lower level access to advanced controls (aws s3api). The plumbing command aws s3api get-object doesn't support stdout natively, so if you need it and want to use it in a pipeline (e.g. pv), you would naively do something like
Unfortunately, aws s3api already prints the API response to stdout, and error messages to stderr, so if you do the above you'll clobber your pipeline with noise, and using /dev/stderr has the same effect on error.
The existing build system I did not have control over, and would produce output on stdout/stderr. I wanted my build scripts to be able to only show the output from the build system if building failed (and there might have been multiple build system invocations leading to that failure). I also wanted the second level to be able to log progress messages that were shown to the user immediately on stdout.
Level 1: create fd=3, capture fd 1/2 (done in one place at the top-level)
Level 2: log progress messages to fd=3 so the user knows what's happening
Level 3: original build system, will log to fd 1/2, but will be captured
It was janky and it's not a project I have a need for anymore, but it was technically a real world use case.
It was a while ago since I implemented this, but iirc the reason for that was to validate that the key that has signed this is actually trusted, and the signature isn't just cryptographically valid.
You can also redirect specific file descriptors into other commands:
The aws cli has a set of porcelain for s3 access (aws s3) and plumbing commands for lower level access to advanced controls (aws s3api). The plumbing command aws s3api get-object doesn't support stdout natively, so if you need it and want to use it in a pipeline (e.g. pv), you would naively do something like
Unfortunately, aws s3api already prints the API response to stdout, and error messages to stderr, so if you do the above you'll clobber your pipeline with noise, and using /dev/stderr has the same effect on error.
You can, though, do the following:
This will pipe only the object contents to stdout, and the API response to /dev/null.
Would be nice if `curl` had something to dump headers to a third file descriptor while outputting the response on stdout.
This should work?
or
unless I'm misunderstanding you.
1 reply →
I have used this in the past when building shell scripts and Makefiles to orchestrate an existing build system:
https://github.com/jez/symbol/blob/master/scaffold/symbol#L1...
The existing build system I did not have control over, and would produce output on stdout/stderr. I wanted my build scripts to be able to only show the output from the build system if building failed (and there might have been multiple build system invocations leading to that failure). I also wanted the second level to be able to log progress messages that were shown to the user immediately on stdout.
It was janky and it's not a project I have a need for anymore, but it was technically a real world use case.
One of my use-cases previously has been enforcing ultimate or fully trust of a gpg signature.
It was a while ago since I implemented this, but iirc the reason for that was to validate that the key that has signed this is actually trusted, and the signature isn't just cryptographically valid.
You can also redirect specific file descriptors into other commands:
Red hat and other RPM based distributions recommended kickstart scripts use tty3 using a similar method
Multiple levels of logging, all of which you want to capture but not all in the same place.
Wasn't the idiomatic way the `-v` flag (repeated for verbosity). And then stderr for errors (maybe warning too).
It is, and all logs should ideally go to stderr. But that doesn’t let you pipe them to different places.