Comment by gchamonlive
19 days ago
The "just keep your CI simple" mindset doesn't work in practice. Any non-trivial project will have a high chance that it'll have to encode some form of logic in the CI, either for situational triggers, or git branching strategies, on demand deployments, permissions, secrets, heterogeneous runners, load balance, local testing, component testing... these are all valid use-cases, all with their own gotchas and hard-to-debug issues in all CI systems I know.
It's correct to design CI pipelines in order to offload much of the logic to subsystems, but pipelines will eventually grow in complexity and the CI config system should be designed in order not to get in the way. I don't know buildkite, but Gitlab CI is the best I know. Template and job composition works brilliantly, top-level object being the job and not the stage result in flat, easier to read config files and the packed features are really good, but it's hard to debug, the conditional logic sometimes fails in unexpected ways, it's exhausting to use the predefined variables reference and the permission system for multi project pipelines is abysmal.
I don't think we're necessarily in disagreement - your points about reusing CI code across jobs through templating or composition are well taken.
I'd argue that this also dovetails very nicely with having common, shared invocations - if you can run "make test" in any repo and have it work, that makes CI code reuse even easier.
As for the complexity comments, that complexity has to go somewhere, and you should look for how to best factor the system so it's debuggable. Sometimes this may mean restructuring how your code is factored or deployed or has failure tolerance so it's easier to test, and this should be thought of as an architecture task early on.