Comment by deathanatos

19 hours ago

You're not wrong, but at my place, our main repository does not permit cloning into a directory with spaces in it.

Three factors conspire to make a bug:

  1. Someone decides to use a space
  2. We use Python
  3. macOS

Say you clone into a directory with a space in it. We use Python, so thus our scripts are scripts in the Unix sense. (So, Python here is replacable with any scripting language that uses a shebang, so long as the rest of what comes after holds.) Some of our Python dependencies install executables; those necessarily start with a shebang:

  #!/usr/bin/env python3

Note that space.

Since we use Python virtualenvs,

  #!/home/bob/src/repo/.venv/bin/python3

But … now what if the dir has a space?

  #!/home/bob/src/repo with a space/.venv/bin/python3

Those look like arguments, now, to a shebang. Shebangs have no escaping mechanism.

As I also discovered when I discovered this, the Python tooling checks for this! It will instead emit a polyglot!

  #!/bin/bash

  # <what follows in a bash/python polyglot>
  # the bash will find the right Python interpreter, and then re-exec this
  # script using that interpreter. The Python will skip the bash portion,
  # b/c of cleverness in the polyglot.

Which is really quite clever, IMO. But, … it hits (2.). It execs bash, and worse, it is macOS's bash, and macOS's bash will corrupt^W remove for your safety! certain environment variables from the environment.

Took me forever to figure out what was going on. So yeah … spaces in paths. Can't recommend them. Stuff breaks, and it breaks in weird and hard to debug ways.

4 comments

deathanatos

joshuaissac 19 hours ago

If all of your scripts run in the same venv (for a given user), can you inject that into the PATH and rely on env just finding the right interpreter?

I suppose it would also need env to be able to handle paths that have spaces in them.

ndsipa_pomu 13 hours ago

What a headache!

My practical view is to avoid spaces in directories and filenames, but to write scripts that handle them just fine (using BASH - I'm guilty of using it when more sane people would be using a proper language).

My ideological view is that unix/POSIX filenames are allowed to use any character except for NULL, so tools should respect that and handle files/dirs correctly.

I suppose for your usage, it'd be better to put the virtualenv directory into your path and then use #!/usr/bin/env python

skydhash 9 hours ago

For the BSDs and Linux, I believe that shebang are intepreted by the kernel directly and not by the shell. /usr/bin/env and /bin/sh are guaranteed by POSIX to exists so your solution is the correct one. Anything else is fragile.

balnaphone 9 hours ago

These are part of the rituals of learning how a system works, in the same way interns get tripped up at first when they discover ^S will hang an xterm, until ^Q frees it. If you're aware of the history of it, it makes perfect sense. Unix has a personality, and in this case the kernel needs to decide what executable to run before any shell is involved, so it deliberately avoids the complexity of quoting rules.

I'd give this a try, works with any language:

  #!/usr/bin/env -S "/path/with spaces/my interpreter" --flag1 --flag2

Only if my env didn't have -S support, I might consider a separate launch script like:

  #!/bin/sh
  exec "/path/with spaces/my interpreter" "$0" "$@"

But most decent languages seems to have some way around the issue.

Python

  #!/bin/sh
  """:"
  exec "/path/with spaces/my interpreter" "$0" "$@"
  ":"""
  # Python starts here
  print("ok")

Ruby

  #!/bin/sh
  exec "/path/with spaces/ruby" -x "$0" "$@"
  #!ruby
  puts "ok"

Node.js

  #!/bin/sh
  /* 2>/dev/null
  exec "/path/with spaces/node" "$0" "$@"
  */
  console.log("ok");

Perl

  #!/bin/sh
  exec "/path/with spaces/perl" -x "$0" "$@"
  #!perl
  print "ok\n";

Common Lisp (SBCL) / Scheme (e.g. Guile)

  #!/bin/sh
  #|
  exec "/path/with spaces/sbcl" --script "$0" "$@"
  |#
  (format t "ok~%")

  #!/bin/sh
  #if 0
  exec "/path/with spaces/tcc" -run "$0" "$@"
  #endif
  
  #include <stdio.h>
  
  int main(int argc, char **argv)
  {
      puts("ok");
      return 0;
  }

Racket

  #!/bin/sh
  #|
  exec "/path/with spaces/racket" "$0" "$@"
  |#
  #lang racket
  (displayln "ok")

Haskell

  #!/bin/sh
  #if 0
  exec "/path/with spaces/runghc" -cpp "$0" "$@"
  #endif
  
  main :: IO ()
  main = putStrLn "ok"

Ocaml (needs bash process substitution)

  #!/usr/bin/env bash
  exec "/path/with spaces/ocaml" -no-version /dev/fd/3 "$@" 3< <(tail -n +3 "$0")
  print_endline "ok";;