Comment by garethrowlands

2 days ago

Articles like this are fun but they all come from posix shell syntax being fundamentally bad for scripting/programming. All the piping stuff is great, of course. And the overall ecosystem is great. But the interpretation of the script itself working by a series of string substitutions is a mechanism we wouldn't accept in a regular programming language. And there's no excuse for it really, except that shell syntax is really, really old.

For example, what does `$foo` mean in shell syntax? In any reasonable language (perl or powershell, for example, or python if you drop the `$`), it's an expression that evaluates to whatever value's inside that variable. In shell, `$foo` isn't an expression in that sense, and what it does depends on what's inside it via a variety of string substitution rules.

This is the main reason we have arcane articles like this.

That said, nice article.

I feel the need to push back on this perspective.

Old doesn't imply arcane. It instead can (and does, in this case) mean that it won out over the course of decades against other less worthy alternatives.

Although you can write "programs" in sh, the shell syntax was never meant to be anything like a systems or application programming language. It was meant to efficiently automate systems tasks. It is kind of like the fork lift of computing. It does it's job very well, is bad at doing anything else, yet no other kind of vehicle can do it's job nearly as efficiently.

Comparing the shell to C, Go, Rust, Python, or JavaScript is crazypants. It has a different job than those, so of course it will look and feel different!

  • There are plenty of alternative shells that don't suffer these problems. Posix syntax didn't win because it was better - it's the classic 'worse is better'.

    > Old doesn't imply arcane.

    Indeed so. For example, awk is about the same age but doesn't suffer the same problems.

    > the shell syntax was never meant to be anything like a systems or application programming language

    A programmable shell is not only like a programming language, it is a programming language. Systems tasks have some specialised requirements but, overall, scripting is programming. There is nothing in the problem domain that requires the posix string-substitution problems.

    If fish or oil had existed back then, they would likely have won, since they are better.

    • > overall, scripting is programming.

      This is a logical statement that is technically true while not matching real-world experience. Scripting as a type of programming is different from application development programming, and scripting needs differences in features and interface to be convenient and comfortable.

      > There is nothing in the problem domain that requires the posix string-substitution problems.

      What do you mean? Environment variables and command line execution both separately lead to string substitutions. The fish shell uses string substitutions.

      You emphasized ‘requires’ which again might make your statement technically true while still missing the experiential forest for the logical trees. You can use x86 assembly for your shell if you want, there’s nothing that “requires” posix. The reason string substitutions exist is because they’re useful and convenient; the alternatives are verbose and klunky.

    • I think the point is that worse is better. There is incredible power in a standard (mostly) predictable platform, even if it could be done better.

  • I think Bash won as Bityard says above, in the sense that COBOL won, which was because it was everywhere (approximately) and then when another system was being built to do something in COBOL’s niche COBOL was the obvious choice. Once it was sufficiently established it won by inertia and absolutely not because it was the better language. Or maybe it was the better language in its time, but it certainly is not now. And note I’m only talking about COBOL’s niche, of mainframe-based processing in the banking, insurance, state and federal payroll and similar areas.

    • In a similar way that Python won, despite having some pretty delulu characteristics.

      Whitespace sensitivity is horrendous (can’t autoformat, no multi-line lambdas, bonkers inline list comprehensions) but here we are.

  • > It was meant to efficiently automate systems tasks.

    Shell was designed to launch processes easily. Unix philosophy pushed to make every function an application. And that's ultimately what shell "syntax" is.

    There's shockingly little syntax in the shell. A lot of what we think of as "shell syntax" is actually full blown (standard) programs which the shell is launching. My favorite of which is `[`.

    What makes shell unique vs all other programming languages is it puts launching processes first. It has no friction there because if it did, it wouldn't work.

    This is also what I think is gross about shell. Every other scripting language has at least some friction when launching other processes and that's usually a good thing IMO.

  • One of these are not like the other. Python is perfectly serviceable as a shell, to the extent that one could largely replace bash with the Python REPL if they wanted.

    • python you say?

      Pray tell which one of the myriad is the true one?

      And do not worry, no one would be rash enough to bring in external dependencies it is just too darn inconvenient.

      I should probably drink my coffee before posting, be less grumpy.

      3 replies →

    • I'd be jumping out the top storey window after about 2 minutes of being required to use Python as a shell.

      Python is a full-on programming language. Its REPL is a REPL for that language. Interacting with the OS and filesystem is a niche, tucked away in a corner of the language.

      It has has none of the ergonomic affordances needed to be the user interface for interacting with files and processes.

      Here are some simple commands:

          ls
          cd foo
          ls
          rm bar
          cd ../baz
          rm bar
          echo >readme.txt baz directory
          curl -s http://example.com >download
      

      Let's try doing that with python:

          python
          >>> ls
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          NameError: name 'ls' is not defined
          >>> import system
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          ModuleNotFoundError: No module named 'system'
          >>> import os
          >>> os.dir.list()
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          AttributeError: module 'os' has no attribute 'dir'
          >>> help(os)
      
          >>> os.listdir()
          ['foo', 'bar', 'baz']
          >>> cd foo
            File "<stdin>", line 1
              cd foo
                 ^
          SyntaxError: invalid syntax
          >>> chdir("foo")
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          NameError: name 'chdir' is not defined
          >>> os.chdir("foo")
          >>> os.listdir()
          ['bar']
          >>> os.delete("bar")
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          AttributeError: module 'os' has no attribute 'delete'
          >>> os.remove("bar")
          >>> os.chdir("../baz")
          >>> os.remove("bar")
          >>> f = open("readme.txt", "w")
          >>> f.write("baz directory")
          13
          >>> f.close()
          >>> import process
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          ModuleNotFoundError: No module named 'process'
          >>> import subprocess
          >>> subprocess.run("curl", "-s", "http://example.com/")
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
            File "/usr/lib/python3.9/subprocess.py", line 505, in run
              with Popen(*popenargs, **kwargs) as process:
            File "/usr/lib/python3.9/subprocess.py", line 778, in __init__
              raise TypeError("bufsize must be an integer")
          TypeError: bufsize must be an integer
          >>> subprocess.run(["curl", "-s", "http://example.com/"])
          ...
          CompletedProcess(args=['curl', '-s', 'http://example.com/'], returncode=0)
          >>> subprocess.run(args=["curl", "-s", "http://example.com/"], capture_output=True)
          CompletedProcess(args=['curl', '-s', 'http://example.com/'], returncode=0, stdout=b'...\n', stderr=b'')
          >>> p = subprocess.run(args=["curl", "-s", "http://example.com/"], capture_output=True)
          >>> fh = open("download", "w")
          >>> fh.write(p.stdout)
          Traceback (most recent call last):
            File "<stdin>", line 1, in <module>
          TypeError: write() argument must be str, not bytes
          >>> fh = open("download", "wb")
          >>> fh.write(p.stdout)
          559
          >>> fh.close()

      4 replies →

> fundamentally bad for scripting/programming […] a mechanism we wouldn’t accept in a regular programming language

I don’t agree with this assumption that shell should use mechanisms we accept in “regular” programming languages, whatever that means. And why not the other way around? Why do we accept mechanisms in programming that we wouldn’t accept in a shell? There are layers of assumption in your assumption.

I also disagree with lumping scripting and programming together. Those are two different activities, and shell is better at scripting than programming, and it’s better at scripting than programming languages.

Try using python as your shell, and you’ll find out that python is fundamentally bad as a shell REPL. The shell is necessarily different from python or C++ or <your_favorite_programming_language>, starting with the ability to run command lines by typing them, with zero extra syntax. That is something regular programming languages can’t do, and it leads directly to the need for string substitution rules.

If you want a shell made out of a good programming language… make one! People have been thinking about this for decades and nobody has come up with a good one, because shell languages like bash are better at shelling than programming languages.

  • a bunch of happy xonsh users disagree

    • Hey good! I’d like a good python shell. Does xonsh fix cut and paste with python? The first huge problem I ran into with python shelling is you can’t cut as paste indented code at all.

      Edit I’m reading about it, here are some notes to myself:

      Sounds like there are 2 modes, subprocess & python

      Env vars are still string substitution, and there are many rules and string literal prefixes.

      You will want the prompt toolkit (ptk). It’s an additional install. (Non-default installs, btw, are why I stopped using zsh. I loved zsh, but had to move to bash.)

      Regexes use backticks

      @.imp for inline imports… good idea

      BTW I googled for xonsh complaints and got what I expected; awkward & klunky interactive editing, special prefix syntax, some missing features, system python dependency issues, high CPU usage, and lack of portability. People seem to suggest that xonsh is better for scripting than interactive usage.

I disagree. I see shell syntax as an “inversion” of normal quoting rules. Normally quoting is the default but to achieve anything special you have to use an escape. Conversely, in shell, quoting is not necessary but any special character will likely have unwanted behaviour unless you quote.

This is handy for quick scripting as a natural extension of a command line interface. A counterexample might be an equivalent script using the Python subprocess module. For a simple script, the required extra quoting overhead is not worth the trouble. For something complicated it’s essential and a shell script is no longer suitable. For something in the middle, a carefully crafted shell script (eg. one that passes shellcheck) may be the most suitable solution depending on the situation.

Both cases have their place. It’s about using the right tool for the job.

I will immediately remove all the shell scripts on my system based on this comment alone.

Posted from my phone since my system came crashing down.

Yes a large percentage of bash scripts fails if you feed them filenames with spaces or other special characters. Or a large amount of filenames near the Unix maximum commandline length.

Bash et al are great for command line use, but produce a situation of really bad engineering hygiene when used in scripts.

Do not use. Stay away. Bash et al considered harmful. Red flags on job interview when referenced.

  • Whilst I agree that Bash makes it far too easy to do it wrong, it's not too tricky to write scripts that don't have those issues.

    Possibly the most important tip is to use ShellCheck (https://www.shellcheck.net/) as a linter on your scripts and work through the various warnings to eliminate them. For the rare times that ShellCheck is overly cautious, you can remove a specific warning with a comment before the offending line such as

      # shellcheck disable=SC1091
    

    Also, take time to look through Greg's Wiki as it is possibly the best Bash resource that discusses the various pitfalls: https://mywiki.wooledge.org/BashPitfalls

    As a Bash script writer, I disagree with your "red flag" interpretation as that'll mean that you'll end up with poorly written Bash scripts along with a hotch-potch of different tools to achieve the functionality (e.g. different versions of Python required for older scripts and newer ones). Knowing Bash is also very useful for writing Dockerfiles.