← Back to context

Comment by nrclark

4 days ago

Agreed with this sentiment, but with one minor modification: use a Makefile instead. Recipes are still chunks of shell, and they don’t need to produce or consume any files if you want to keep it all task-based. You get tab-completion, parallelism, a DAG, and the ability to start anywhere on the task graph that you want.

It’s possible to do all of this with a pure shell script, but then you’re probably reimplementing some or all of the list above.

9 comments

nrclark

Reply
DrBazza  4 days ago

> use a Makefile instead

I was making a general comment that your build should be a single 'command'. Personally, I don't care what the command is, only that it should be a) one command, and b) 100% runnable on a dev box or a server. If you use make, you'll soon end up writing... shell scripts, so just use a shell script.

In an ideal world your topmost command would be a build tool:

     ./gradlew build
     bazel build //...
     make debug
     cmake --workflow --preset

Unfortunately, the second you do that ^^^, someone edits your CI/CD to add a step before the build starts. It's what people do :(

All the cruft that ends up *in CI config*, should be under version control, and inside your single command, so you can debug locally.

  • chubot  4 days ago

    That's exactly why the "main" should be shell, not make (see my sibling reply). So when someone needs to add that step, it becomes:

        #!/bin/sh

    step-I-added-to-shell-rather-than-CI-yaml
    make debug  # or cmake, bazel

    This is better so you can run the whole thing locally, and on different CI providers

    In general, a CI is not a DAG, and not completely parallel -- but it often contains DAGs

chubot  4 days ago

Make is not a general purpose parallel DAG engine. It works well enough for small C projects and similar, but for problems of even medium complexity, it falls down HARD

Many years ago, I wrote 3 makefiles from scratch as an exploration of this (and I still use them). I described the issues here: https://lobste.rs/s/yd7mzj/developing_our_position_on_ai#c_s...

---

The better style is in a sibling reply -- invoke Make from shell, WHEN you have a problem that fits Make.

That is, the "main" should be shell, not Make. (And it's easy to write a dispatcher to different shell functions, with "$@", sometimes called a "task file" )

In general, a project's CI does not fit entirely into Make. For example, the CI for https://oils.pub/ is 4K lines of shell, and minimal YAML (portable to Github Actions and sourcehut).

https://oils.pub/release/latest/pub/metrics.wwz/line-counts/...

It invokes Make in a couple places, but I plan to get rid of all the Make in favor of Python/Ninja.

  • DrBazza  4 days ago

    Portability to other CI/CDs systems is an understated reason to use a single build command.

dgfitz  4 days ago

You invoke CMake/qmake/configure/whatever from the bash script.

I hate committing makefiles directly if it can be helped.

You can still call make in the script after generating the makefile, and even pass the make target as an argument to the bash script if you want. That being said, if you’re passing more than 2-3 arguments to the build.sh you’re probably doing it wrong.

  • nrclark  4 days ago

    Yes to calling CMake/etc. No to checking in generated Makefiles. But for your top-level “thing that calls CMake”, try writing a Makefile instead of a shell script. You’ll be surprised at how powerful it is. Make is a dark horse.

    • dgfitz  4 days ago

      I wouldn't be surprised at all, make is great!

      My contention is that a build script should ideally be:

      sha-bang

      clone && cd $cloned_folder

      ${generate_makefile_with_tool}

      make $1

      Anything much longer than that can (and usually will) quickly spiral out of control.

      Make is great. Unless you're code-golfing, your makefile will be longer than a few lines and a bunch of well-intentioned-gremlins will pop in and bugger the whole thing up. Just seen it too many times.

      Edit: in the jenkins case, in a jenkins build shell the clone happens outside build.sh:

      (in jenkins shell):

      clone && cd clone ./build.sh $(0-1 args)

      (inside build.sh): $(generate_makefile_with_tool) make $1

  • lenkite  3 days ago

    I have experienced horror build systems where the Makefile delegates to a shell script which then delegates to some sub-module Makefile, which then delegates to a shell script...

    The problem is that shell commands are very painful to specify in a Makefile with weird syntactical rules. Esp when you need them to run in one shell - a lot of horror quoting needed.