Comment by 1a527dd5
11 hours ago
1. Don't use bash, use a scripting language that is more CI friendly. I strongly prefer pwsh.
2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts.
3. Having standalone scripts will allow you to develop/modify and test locally without having to get caught in a loop of hell.
4. Design your entire CI pipeline for easier debugging, put that print state in, echo out the version of whatever. You don't need it _now_, but your future self will thank you when you do it need it.
5. Consider using third party runners that have better debugging capabilities
I would disagree with 1. if you need anything more than shell that starts to become a smell to me. The build/testing process etc should be simple enough to not need anything more.
That's literally point #2, but I had the same reaction as you when I first read point #1 :)
I agree with #2, I meant more if you are calling out to something that is not a task runner(Make, Taskfile, Just etc) or a shell script thats a bit of a smell to me. E.g. I have seen people call out to Python scripts etc and it concerns me.
30 replies →
I mean, at some point you are bash calling some other language anyway.
I'm a huge fan of "train as you fight", whatever build tools you have locally should be what's used in CI.
If your CI can do things that you can't do locally: that is a problem.
> If your CI can do things that you can't do locally: that is a problem.
IME this is where all the issues lie. Our CI pipeline can push to a remote container registry, but we can't do this locally. CI uses wildly different caching strategies to local builds, which diverges. Breaking up builds into different steps means that you need to "stash" the output of stages somewhere. If all your CI does is `make test && make deploy` then sure, but when you grow beyond that (my current project takes 45 minutes with a _warm_ cache) you need to diverge, and that's where the problems start.
2 replies →
> If your CI can do things that you can't do locally: that is a problem.
Probably most of the times when this is an actual problem, is building across many platforms. I'm running Linux x86_64 locally, but some of my deliverables are for macOS and Windows and ARM, and while I could cross-compile for all of them on Linux (macOS was a bitch to get working though), it always felt better to compile on the hardware I'm targeting.
Sometimes there are Windows/macOS-specific failures, and if I couldn't just ssh in and correct/investigate, and instead had to "change > commit > push" in an endless loop, it's possible I'd quite literally would lose my mind.
7 replies →
> If your CI can do things that you can't do locally: that is a problem.
Completely agree.
> I'm a huge fan of "train as you fight", whatever build tools you have locally should be what's used in CI.
That is what I am doing, having my GitHub Actions just call the Make targets I am using locally.
> I mean, at some point you are bash calling some other language anyway.
Yes, shell scripts and or task runners(Make, Just, Task etc) are really just plumbing around calling other tools. Which is why it feels like a smell to me when you need something more.
I don't agree with (1), but agree with (2). I recommend just putting a Makefile in the repo and have that have CI targets, which you can then easily call from CI via a simple `make ci-test` or similar. And don't make the Makefiles overcomplicated.
Of course, if you use something else as a task runner, that works as well.
For certain things, makefiles are great options. For others though they are a nightmare. From a security perspective, especially if you are trying to reach SLSA level 2+, you want all the build execution to be isolated and executed in a trusted, attestable and disposable environment, following predefined steps. Having makefiles (or scripts) with logical steps within them, makes it much, much harder to have properly attested outputs.
Using makefiles mixes execution contexts between the CI pipeline and the code within the repository (that ends up containing the logic for the build), instead of using - centrally stored - external workflows that contains all the business logic for the build steps (e.g., compiler options, docker build steps etc.).
For example, how can you attest in the CI that your code is tested if the workflow only contains "make test"? You need to double check at runtime what the makefile did, but the makefile might have been modified by that time, so you need to build a chain of trust etc. Instead, in a standardized workflow, you just need to establish the ground truth (e.g., tools are installed and are at this path), and the execution cannot be modified by in-repo resources.
That doesn't make any sense. Nothing about SLSA precludes using make instead of some other build tool. Either inputs to a process are hermetic and attested or they're not. Makefiles are all about executing "predefined steps".
It doesn't matter whether you run "make test" or "npm test whatever": you're trusting the code you've checked out to verify its own correctness. It can lie to you either way. You're either verifying changes or you're not.
4 replies →
Makefile or scripts/do_thing either way this is correct. CI workflows should only do 1 thing each step. That one thing should be a command. What that command does is up to you in the Makefile or scripts. This keeps workflows/actions readable and mostly reusable.
>I don't agree with (1)
Neither do most people, probably but it's kinda neat how they suggested fix for github actions' ploy to maintain vendor lock-in is to swap it with a language invented by that very same vendor.
makefile commands are the way
I was once hired to manage a build farm. All of the build jobs were huge pipelines of Jenkins plugins that did various things in various orders. It was a freaking nightmare. Never again. Since then, every CI setup I’ve touched is a wrapper around “make build” or similar, with all the smarts living in Git next to the code it was building. I’ll die on this hill.
#2 is not a slam dunk because the CI system loses insight into your build process if you just use one big script.
Does anyone have a way to mark script sections as separate build steps with defined artifacts? Would be nice to just have scripts with something like.
They could noop on local runs but be reflected in the github/gitlab as separate steps/stages and allow resumes and retries and such. As it stands there's no way to really have CI/CD run the exact same scripts locally and get all the insights and functionality.
I haven't seen anything like that but it would be nice to know.
Do you (or does anyone) see possible value in a CI tool that just launches your script directly?
It seems like if you
> 2. Don't have logic in your workflows. Workflows should be dumb and simple (KISS) and they should call your scripts.
then you’re basically working against or despite the CI tool, and at that point maybe someone should build a better or more suitable CI tool.
Can we have a CI tool, that simply takes a Makefile as input? Perhaps takes all targets, that start with "ci" or something.
Build a CLI in python or whatever which does the same thing as CI, every CI stage should just call its subcommands.
Just use a task runner(Make, Just, Taskfile) this is what they were designed for.
I typically use make for this and feel like I’m constantly clawing back scripts written in workflows that are hard to debug if they’re even runnable locally.
This isn’t only a problem with GitHub Actions though. I’ve run into it with every CI runner I’ve come across.
In many enterprise environments, deployment logic would be quite large for bash.
1 reply →
How do you handle persistent state in your actions?
For my actions, the part that takes the longest to run is installing all the dependencies from scratch. I'd like to speed that up but I could never figure it out. All the options I could find for caching deps sounded so complicated.
> How do you handle persistent state in your actions?
You shouldn't. Besides caching that is.
> All the options I could find for caching deps sounded so complicated.
In reality, it's fairly simple, as long as you leverage content-hashing. First, take your lock file, compute the sha256sum. Then check if the cache has an artifact with that hash as the ID. If it's found, download and extract, those are your dependencies. If not, you run the installation of the dependencies, then archive the results, with the ID set to the hash.
It really isn't more to it. I'm sure there are helpers/sub-actions/whatever Microsoft calls it, for doing all of this with 1-3 lines or something.
The tricky bit for me was figuring out which cache to use, and how to use and test it locally. Do you use the proprietary github actions stuff? If the installation process inside the actions runner is different from what we use in the developer machines, now we have two sets of scripts and it's harder to test and debug...
1 reply →
Depends on the build toolchain but usually you'd hash the dependency file and that hash is your cache key for a folder in which you keep your dependencies. You can also make a Docker image containing all your dependencies but usually downloading and spinning that up will take as long as installing the dependencies.
For caching you use GitHubs own cache action.
You don't.
For things like installing deps, you can use GitHub Actions or several third party runners have their own caching capabilities that are more mature than what GHA offers.
If you are able to use the large runners, custom images are a recent addition to what Github offers.
https://docs.github.com/en/actions/how-tos/manage-runners/la...
Minor variance on #1, I've come to use Deno typescripts for anything more complex than what can be easily done in bash or powershell. While I recognize that pwsh can do a LOT in the box, I absolutely hate the ergonomics and a lot of the interactions are awkward for people not used to it, while IMO more developers will be more closely aligned to TypeScript/JavaScript.
Not to mention, Deno can run TS directly and can reference repository/http modules directly without a separate install step, which is useful for shell scripting beyond what pwsh can do. ex: pulling a dbms client and interacting directly for testing, setup or configuration.
For the above reasons, I'll also use Deno for e2e testing over other languages that may be used for the actual project/library/app.
> Don't use bash
What? Bash is the best scripting language available for CI flows.
1. Just no. Unless you are some sort of Windows shop.
Pwsh scripts are portable across mac, linux and windows with arguably less headache than bash. Its actually really nice. You should try it.
If you don't like it, you can get bash to work on windows anyway.
If you're building for Windows, then bash is "just no", so it's either cmd/.bat, or pwsh/.ps. <shrugs>
All my windows work / ci runs still use bash.
I develop on Windows. And I use bash and (gnu) make - combination that cannot be beat, in my experience.
That’s the only reason for sure.
I mean, if you're a Windows shop you really should be using powershell.
Step 0. Stop using CI services that purposefully waste your time, and use CI services that have "Rebuild with SSH" or similar. From previous discussions (https://news.ycombinator.com/item?id=46592643), seems like Semaphore CI still offers that.