← Back to context

Comment by jez

1 day ago

Another fun consequence of this is that you can initialize otherwise-unset file descriptors this way:

    $ cat foo.sh
    #!/usr/bin/env bash

    >&1 echo "will print on stdout"
    >&2 echo "will print on stderr"
    >&3 echo "will print on fd 3"

    $ ./foo.sh 3>&1 1>/dev/null 2>/dev/null
    will print on fd 3

It's a trick you can use if you've got a super chatty script or set of scripts, you want to silence or slurp up all of their output, but you still want to allow some mechanism for printing directly to the terminal.

The danger is that if you don't open it before running the script, you'll get an error:

    $ ./foo.sh
    will print on stdout
    will print on stderr
    ./foo.sh: line 5: 3: Bad file descriptor

18 comments

jez

Reply
hielke  11 hours ago

With exec you can open file descriptors of your current process.

  if [[ ! -e /proc/$$/fd/3 ]]; then
      # check if fd 3 already open and if not open, open it to /dev/null
      exec 3>/dev/null
  fi
  >&3 echo "will print on fd 3"

This will fix the error you are describing while keeping the functionality intact.

Now with that exec trick the fun only gets started. Because you can redirect to subshells and subshells inherit their redirection of the parent:

  set -x # when debugging, print all commands ran prefixed with CMD:
  PID=$$
  BASH_XTRACEFD=7
  LOG_FILE=/some/place/to/your/log/or/just/stdout
  exec 3> >(gawk '!/^RUN \+ echo/{ print strftime("[%Y-%m-%d %H:%M:%S] <PID:'$PID'> "), $0; fflush() }' >> $LOG_FILE)
  exec > >(sed -u 's/^/INFO:  /' >&3)
  exec 2> >(sed -u 's/^/ERROR: /' >&3)
  exec 7> >(sed -u 's/^/CMD:   /' >&3)
  exec 8>&1 #normal stdout with >&8
  exec 9>&2 #normal stderr with >&9

And now your bash script will have a nice log with stdout and stderr prefixed with INFO and ERROR and has timestamps with the PID.

Now the disclaimer is that you will not have gaurantees that the order of stdout and stderr will be correct unfortunately, even though we run it unbuffered (-u and fflush).

  • casey2  6 hours ago

    Nice! Not really sure the point since AI can bang out a much more maintainable (and sync'd) wrapper in go in about 0.3 seconds

    (if runners have sh then they might as well have a real compiler scratch > debian > alpine , "don't debug in prod")

account42  15 hours ago

If you just want to print of the terminal even if normal stdout/stderr is disabled you can also use >/dev/tty but obviously that is less flexible.

47282847  1 day ago

Interesting. Is this just literally “fun”, or do you see real world use cases?

  • nothrabannosir  1 day ago

    The aws cli has a set of porcelain for s3 access (aws s3) and plumbing commands for lower level access to advanced controls (aws s3api). The plumbing command aws s3api get-object doesn't support stdout natively, so if you need it and want to use it in a pipeline (e.g. pv), you would naively do something like

      $ aws s3api get-object --bucket foo --key bar /dev/stdout | pv ...

    Unfortunately, aws s3api already prints the API response to stdout, and error messages to stderr, so if you do the above you'll clobber your pipeline with noise, and using /dev/stderr has the same effect on error.

    You can, though, do the following:

      $ aws s3api get-object --bucket foo --key bar /dev/fd/3 3>&1 >/dev/null | pv ...

    This will pipe only the object contents to stdout, and the API response to /dev/null.

  • jez  1 day ago

    I have used this in the past when building shell scripts and Makefiles to orchestrate an existing build system:

    https://github.com/jez/symbol/blob/master/scaffold/symbol#L1...

    The existing build system I did not have control over, and would produce output on stdout/stderr. I wanted my build scripts to be able to only show the output from the build system if building failed (and there might have been multiple build system invocations leading to that failure). I also wanted the second level to be able to log progress messages that were shown to the user immediately on stdout.

        Level 1: create fd=3, capture fd 1/2 (done in one place at the top-level)
    Level 2: log progress messages to fd=3 so the user knows what's happening
    Level 3: original build system, will log to fd 1/2, but will be captured

    It was janky and it's not a project I have a need for anymore, but it was technically a real world use case.

  • 1718627440  12 hours ago

    This is often used by shell scripts to wrap another program, so that those's input and output can be controlled. E.g. Autoconf uses this to invoke the compiler and also to control nested log output.

  • figmert  18 hours ago

    One of my use-cases previously has been enforcing ultimate or fully trust of a gpg signature.

        tmpfifo="$(mktemp -u -t gpgverifyXXXXXXXXX)"
    gpg --status-fd 3 --verify checksums.txt.sig checksums.txt 3>$tmpfifo
    grep -Eq '^\[GNUPG:] TRUST_(ULTIMATE|FULLY)' $tmpfifo

    It was a while ago since I implemented this, but iirc the reason for that was to validate that the key that has signed this is actually trusted, and the signature isn't just cryptographically valid.

    You can also redirect specific file descriptors into other commands:

        gpg --status-fd 3 --verify checksums.txt.sig checksums.txt 3>(grep -Eq '^\[GNUPG:] TRUST_(ULTIMATE|FULLY)')

  • jas-  1 day ago

    Red hat and other RPM based distributions recommended kickstart scripts use tty3 using a similar method