Why does air-gapped environment require rolling your own CI/CD solution? Plenty of examples of air-gapped Jenkins and/or Argo Workflows. Was this just an educational exercise?
Argo Workflows does not live up to what they advertise, it is so much more complex to setup and then build workflows for. Helm + Argo is pain (both use the same template delimiters...)
Jenkins, like many tools with extreme flexibility, sucks as much as you make it suck. You can pretty easily turn Jenkins into a monstrosity that makes everyone afraid to ever try to update it. On the other hand, you can also have a pretty solid setup that is easy to work on. The trouble is that the tool itself doesn't guide you much to the good path, so unless you've seen a pleasant Jenkins instance before you're likely to have a worse time than necessary.
It seems like a simple CI/CD in an airgapped environment might be simpler to implement than to (1) learn and (2) onboard an off-the-shelf solution when your airgapped requirement limits your ability to leverage the off-the-shelf ecosystem.
Since you're exercising, you can take it to the next level where you don't specify the next step but the inputs to each task, allowing you to infer the DAG and implement caching...
> Build a dependency graph (which jobs need which other jobs)
> Execute jobs in topological order (respecting dependencies)
For what it’s worth, Python has graphlib.TopologicalSorter in the standard library that can do this, including grouping tasks that can be run in parallel:
I like the iterative approach that you took with the post and codebase. It really takes the reader on a journey with you and helps understand the decision points and process of software development, it's so important compared to just showing a final polished solution, after all, we're all trying to replicate the software development process, not the product.
Docker is unusable for build tools that use namespaces (of which Docker itself is one), unless you use privileged mode and throw away much more security than you'd need to. Docker images are difficult to reproduce with conventional Docker tools, and using a non-reproducible base image for your build environment seems like a rather bad idea.
Why does air-gapped environment require rolling your own CI/CD solution? Plenty of examples of air-gapped Jenkins and/or Argo Workflows. Was this just an educational exercise?
Jenkins sucks but is insanely reliable
Argo Workflows does not live up to what they advertise, it is so much more complex to setup and then build workflows for. Helm + Argo is pain (both use the same template delimiters...)
Jenkins, like many tools with extreme flexibility, sucks as much as you make it suck. You can pretty easily turn Jenkins into a monstrosity that makes everyone afraid to ever try to update it. On the other hand, you can also have a pretty solid setup that is easy to work on. The trouble is that the tool itself doesn't guide you much to the good path, so unless you've seen a pleasant Jenkins instance before you're likely to have a worse time than necessary.
7 replies →
It seems like a simple CI/CD in an airgapped environment might be simpler to implement than to (1) learn and (2) onboard an off-the-shelf solution when your airgapped requirement limits your ability to leverage the off-the-shelf ecosystem.
This was more like an educational exercise
Since you're exercising, you can take it to the next level where you don't specify the next step but the inputs to each task, allowing you to infer the DAG and implement caching...
3 replies →
> We need to:
> Build a dependency graph (which jobs need which other jobs)
> Execute jobs in topological order (respecting dependencies)
For what it’s worth, Python has graphlib.TopologicalSorter in the standard library that can do this, including grouping tasks that can be run in parallel:
https://docs.python.org/3/library/graphlib.html
One of the best real "batteries" added in recent years.
I like the iterative approach that you took with the post and codebase. It really takes the reader on a journey with you and helps understand the decision points and process of software development, it's so important compared to just showing a final polished solution, after all, we're all trying to replicate the software development process, not the product.
Why use Docker as a build job execution engine? It seems terribly unsuited for this.
It's widely used among DevOps Engineers so hence I picked Docker as it makes it easier to understand
> terribly unsuited
Care to elaborate? If you already deploy in docker then wouldn't this be nice?
Docker is unusable for build tools that use namespaces (of which Docker itself is one), unless you use privileged mode and throw away much more security than you'd need to. Docker images are difficult to reproduce with conventional Docker tools, and using a non-reproducible base image for your build environment seems like a rather bad idea.