Comment by amarshall
9 hours ago
Or you can just…not quote the tilde. Folks always seem to reflexively quote “strings” in Bash while not realizing that (almost) everything is a string and most strings are not quoted and it would be odd to do it (e.g. no one is doing `"ls" "-a" "foo"`).
Not quoting the tilde doesn't help:
Edit: mjmas points out that I am wrong, because different Calvinball rules apply to variable assignments:
Also in general your advice is very bad advice. In command-line arguments, which is the vast majority of the code of any shell script, you should always quote strings that contain variable expansion unless you want them to be implicitly split on spaces after variable expansion, because the thing you're storing in the variable is not an atomic string but rather a space-separated list.
This is almost never what you actually want, and in the rare cases that you do want to store a list, the shell's implicit space splitting usually breaks on filenames containing spaces. Bash has actual array variables which make it possible, but horrible, to handle this in a first-class way:
This ksh feature is absent in the Bourne shell and in dash, but MirBSD ksh, Bash, and zsh all have it.
In general, whenever you see a $variable $expansion in a shell script outside of double quotes, you should suspect that the shell script will probably fail if your filenames or directory names contain spaces. In very many cases, this results in path injection security vulnerabilities. There are contexts where unquoted $variable $expansion is safe, but they are relatively rare.
However, relevant detail here! One of those safe contexts is actually variable assignment, where as mjmas pointed out in their helpful comment below, unquoted variable expansion is actually perfectly safe:
Bash does it only inside variable assignments:
results in:
Oh, thank you for the correction. I was wrong about that. Calvinball.
I feel like this is common knowledge and should not be worth mentioning. But then it apparently is not common knowledge, as the article proves.
Have I run into this at some point?
I certainly have.
Have I learned to quote better and only where appropriate from it?
I certainly have.
Bourne compatible shells take a while to learn and require some experience. This won't change, but alternatives exist, with their own caveats.
Almost nobody that I've ever worked with knows how the shell works. I send them [the docs][1] but what kind of world would this be if people read the docs? Also, not exactly a page turner.
[1]: https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V...
I learned this stuff from friends, O'Reilly books and endless nights of failure and trial and error. I'm certainly not a super wizard, but I feel like I'm not missing any shell skills to be able to do what I need to do.
[Insert mild rant on kids these days having no attention span any more]
also poorly organized and formatted.
This feature, or at least the syntactic unit in question has a name here "bare words". You also have these in some contexts in Perl and Ruby, YAML, and probably some other languages, idk.
I've also seen this even with people who seem like generally competent shell users. Idrgi
Unless you're running a shell command from python. That was the first time I saw a command string broken down into "string" arguments for every thing like that.
In that case, you're not running a "shell command" from python, you're passing arguments to exec. A shell command would be a string interpreted by the shell, and you'd use that for shell syntax things like having the shell do variable interpolation or redirections as part of executing the command.
[dead]
The trick is to quote explicitly and correctly. “ and ‘ are different.
why would you use smart quotes in a terminal like that?
They probably fell foul of some browser text box auto-correct.
1 reply →
why would anyone use smart quotes ever
11 replies →
People quote both too often and too little.
GENERAL RULE
1. Double-quote dollar sign expressions, and nothing else.
2. Single-quote words with a literal special character, and nothing else.
---
I should point out that the author's example is NOT fixed by different quoting though.
Because tilde expansion only happens at the beginning of the word.
> I should point out that the author's example is NOT fixed by different quoting though.
Sure, but I would say you almost always want to add to the start of the PATH, not to the end.
I used to put it first for convenience, but got increasingly paranoid about something same-named getting slipped into my ~/.local/bin; now it lives at the end.
The Z shell's, C shell's, and others's syntaxes for setting the PATH environment variable via a shell array variable alias also does the tilde expansion.
It's worth noting, also, that the path and manpath settings in login.conf(5) expand leading tildes in individual search path items.
* https://man.freebsd.org/cgi/man.cgi?query=login.conf&sektion...
So putting the addition of things like ~/bin to PATH in /etc/login_conf and ~/.login_conf instead of shell scripts is another way to address it.
It's particularly handy when there are multiple login shells in use.
* http://jdebp.uk./FGA/BSDs-for-Linux-users/login-conf.html
Or just stop using bash. It’s a terrible language to write and has tons of footguns.
For the things shell is good at (running other commands, pipes, and shuffling files), I have yet to find anything even close to as good.
I agree that shells are uniquely good for that but there are plenty of better shells than Bash.
Such as Fish, nushell, Elvish, or the project I help maintain, “murex”
6 replies →
A different shell??? Nushell, fish, powershell, etc.
4 replies →
I did.
Ruby replaced all my shell needs. Almost 25 years ago. I even have a shell written in ruby (it handles both bash-like behaviour as well as ruby code as-is); admittedly it is not quite perfect for everything, but I improve on it steadily. And it works on Windows too, which was one reason I wrote it in the first place (need to have it work via cmd.exe as-is).
Never looked back to shell. It is too awful to use.
Unfortunately half of its badness isn't isn't the shell itself, but the convention of how parameters are passed to processes on Unix systems.
That’s not correct because POSIX passes what is ostensibly an array of strings.
Windows, on the other hand, only passes one string. So it’s up to the application to choose how to handle whitespace, quotation marks, and other nuances with parsing parameters.
Variable expansion in Bash is lazy. But there’s no reason why variables cannot be tokenised so that strings with spaces aren’t treated as multiple parameters. And in fact that’s exactly how some other shells work, such as the one I maintain.
2 replies →
Array of arguments is vastly superior and more secure than every program/runtime inventing a slightly different way of splitting a command string into an array of arguments. No debate. A real problem is the related birth defect in ssh2.
Well, the only other way I can think of that it could be done is the Windows way, whereby you pass the unparsed command line, spaces and all, as a string to the new process. And while this is arguably the cleaner interface, in practice it has meant even worse quote handling, since how -- or even whether -- double quotes are parsed now depends on the probably undocumented process startup code chosen by the program's compiler vendor.
Want to quote a command line that may already contain double quotes, in order to pass it as an argument to some other program? No, you don't. It isn't right to want that.
1 reply →
yeah but... that convention is so much of a security/bug headache
sometimes, its footguns seem worse than javascript...
hope some typescript-like "typed shell" becomes mainstream someday
6 replies →
The widwit answer: "Dont do X" (and nothing more)
The enlightened answer "You should use A, B or C for these reasons"
Even though bash is installed on many systems, I try to encourage people to use better designed shells that have less of these footguns :
Fish (https://fishshell.com/): No implicit word splitting : spaces in variables won't unexpectedly become separate arguments.
Zsh (I use this: https://ohmyz.sh/): Arrays start at 1 by default, but crucially, unquoted variables don't implicitly split into multiple arguments.
Nushell (https://www.nushell.sh/): Passes structured tables and records between commands : avoids fragile parsing of text with awk/grep.
You need to read the manual.
I think the appropriate method by POSIX rules would be:
I may be wrong. If I'm not, that works in any POSIX-compliant shell.
Edit: this appears to work properly with mksh on my phone:
That's a literal ~ in your path, the exact problem the blog post talks about.
Your use doesn't count as a "word", per man bash. Tilde is only expanded to home at the start of a typically whitespace-separated word, and your tilde is in the middle of one.
> If a word begins with an unquoted tilde character (‘~’), all of the characters up to the first unquoted slash (…) are considered a tilde-prefix. (…)
> word A sequence of characters considered as a single unit by the shell. Also known as a token.
My initial syntax does work in dash (and bash), but all these shells appear to rely on expanding ~ inside a word, that your source asserts is not POSIX (which I do not contest).
Perhaps a succinct and compliant expression could be:
That comes at the cost of forking a subsell.
Edit: 2.6.1 Tilde Expansion in the POSIX standard says that ~ may be expanded "following any unquoted <colon>".
https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V...
That being so, the most succinct and compliant version is my first variant, with the colon moved outside the quotes:
Thank you for prompting me to look this up.