Comment by kazinator

2 days ago

> ( : < dataset.json ) && echo YES # is dataset.json readable?

The subshell execution parentheses and the colon are superfluous here, just:

   < dataset.json && echo YES

Redirections do not require a colon command to hang off of, and there is no need to fork a subshell to execute such a command.

> ( : >> result.json ) && echo YES # is result.json writable?

As a go-to idiom for a writability test, it gives me pause. If the file didn't exist, we created a zero-length one. That might be okay if we are going to write to it anyway as the next action.

If we are testing because we intend to overwrite it, why not just "> result.json" (which is by itself an idiom for truncating a file to zero length).

When would we every do this? Maybe before some command which takes the file name as a destination file argument rather than using output redirection, and which performs a lengthy computation before trying to open the file for writing. We can catch the permission error early.

I don't think I've ever coded such a test; normally you just do the operation that writes to the file and let that fail.

In POSIX C, there is a function access() for doing these kinds of tests. But it has a special purpose: it is meant to be used by a setuid root process to perform a permission test as if it were the real user/group (the one which elevated privilege to root). I.e. it's not can we do this operation, but should we do this operation (would we still be allowed, if we dropped privileges back to the original user).

12 comments

kazinator

refp 2 days ago

They are NOT superflous, and all you need to prove it is `zsh` (but there are others that follow suit in similar fashions):

  zsh% echo "hello world" > data
  
  zsh% < data && echo "READABLE"   # <- will print the contents of data
  hello world
  READABLE
  
  zsh% : < data && echo "READABLE" # <- this will not
  READABLE

So, if you want something that "everyone can use" without going into details about the difference between commonly used shells.. you'd use the null-command.

---

and given that we use the null-command, it _WILL_ behave different with or without subshell.. and all you need is `bash --posix` to prove it:

    % cat subshell.sh
    #!/bin/bash --posix
    ( : < missing.json ); echo AFTER    # <- will echo AFTER
    
    % ./subshell.sh
    ./subshell.sh: line 2: missing.json: No such file or directory
    AFTER
    
    % cat no-subshell.sh
    #!/bin/bash --posix
    : < missing.json; echo AFTER        # <- this will not
    
    % ./no-subshell.sh
    ./no-subshell.sh: line 2: missing.json: No such file or directory

    % : ^- apparently.. there is a difference

The output above is not truncated, `no-subshell.sh` will stop executing due to the broken read.

---

One should never trust things just because they are written, but that also applies to comments on HN. Originally when I read your message I actually thought I made a mistake, I was very close to writing an apology comment and adding a note to the blog post, but not close enough - I had to test it again.

I'm thankful for the watchful eyes and scrutiny when reading things online, that's good - keep it up, but your message is factually wrong - on so many levels.

kazinator 1 day ago
> : < missing.json; echo AFTER # <- this will not
This is required by POSIX in the section "Consequences of Shell Errors". If a redirection error occurs in a special built-in command, a non-interactive shell is required to exit.
The bare redirection without : does not have this problem:
< missing.json; echo AFTER
Therefore, in a POSIX-conforming shell, we do not need to wrap it in a subshell.
> but your message is factually wrong - on so many levels.
How many? Can you count the levels? I didn't know that in zsh "< file" dumps to standard output, which is suppressed by :. If I used Zsh, I would know that sort of thing.
brabel 2 days ago
It’s very common that the most upvoted comment on a post is a mistaken correction. People seem to love comments seemingly proving that the post is wrong without actually checking anything. And once it has enough upvotes a big discussion may follow up with personal attacks on the author or other questionable comments, while the people trying to point out that the post actually is accurate get either buried under the critic storm or downvoted to oblivion because people just don’t like the truth. This happens in every controversial topic.
One of the reasons I stopped writing.
- refp 2 days ago
  
  I have never been in this situation before but I do agree with you, and honestly it feels very bad. I saw the comment when it was posted and, perhaps naively, thought "oh well, no one will care - its just another fluff comment".
  Fast-forward to seeing that comment climb up the ranks, posted by a person who has 270x my karma, who seemingly gets upvotes by just.. writing things? no proof? no rationale? nothing?
  Yeah, feels bad. I'm super happy so many are enjoying the article, and this situation can't take too much away from that, but man.. discouraging for sure.
  
  2 replies →
kazinator 2 days ago
zsh% < data > /dev/null && echo READABLE # <- fixes it, sorta
Maybe zsh still reads the entire file and copies it to /dev/null, which would be a severe performance problem for large files.
The colon by itself, without the subprocess, also prevents the dumping to stdout:
zsh% : < data && echo READABLE
However, with the subprocess parentheses, we can redirect the nonexistence diagnostic to /dev/null, which I don't see mentioned or exemplified in the article:
zsh% : < nonexistent 2> /dev/null && echo READABLE zsh: no such file or directory: nonexistent zsh% (: < nonexistent) 2> /dev/null && echo READABLE
The normal way of testing whether something exists and is readable that you would actually use in production script looks more like this:
$ test -r file && echo YES $ [ -r file ] && echo YES
so we are talking about obfuscated coding.
I would say that "if you want something everyone can use", the -r test would be the first candidate.
BTW, why not bring up the strawman of tcsh?
tcsh% < nonexistent && echo READABLE Invalid null command. tcsh% : < nonexistent && echo READABLE nonexistent: No such file or directory.
Yes, if we want an obfuscated readability test which works literally in any shell that is currently still in deployment in systems that permit new scripts to be installed and run, it looks like do need that colon.
However, fish doesn't like &&:
fish> < nonexistent && echo yes fish: Expected a command, but instead found a redirection fish> : < nonexistent && echo yes fish: Unsupported use of '&&'. In fish, please use 'COMMAND; and COMMAND'. : < nonexistent && echo yes ^
Does it support test -r?
fish> test -r nonexistent fish> test -r nonexistent; and echo yes fish> test -r /etc/hosts; and echo yes yes
Not parentheses though:
fish> ( test -r /etc/hosts ); and echo yes fish: Illegal command name “( test -r /etc/hosts )”
We clearly have to restrict the idea of what "everyone can use" to POSIX-like shells; there is no getting around shell differences absolutely, other than for perhaps trivial command invocations.
- refp 2 days ago
  
  I think you should take a deep breath and read this updated FAQ:
  https://refp.se/articles/your-shell-and-the-magic-colon#why-...
  I am not asking anyone to put this in production, it is a tongue-in-cheek/cute way of explaining/showing what the null-command _can_ do, not what it _must_ do.
  No, any distro that stopped shipping entities mentioned to use only null-command variants.. well, I'd save whatever your end goal is to a meeting with them.
  Just enjoy the article for what it is — a piece of trivia.
  Best Regards, Filip Roséen - refp
  
  1 reply →