Comment by marcosdumay
5 years ago
You rebuild your build cluster with what image? Where do the binaries there come from? And what machine rebuilds the machines.
Securing the machines themselves is a process of adding up always decreasing marginal gains until you say "enough", but the asymptote is never towards a fully secure cluster. That ceiling on how secure you can get is clearly suboptimal.
Besides, the ops people's personal machines have a bunch of high access permissions that can permanently destroy any security you can invent. That isn't any less true if your ops people work for Microsoft instead of you.
I mentioned "github actions" for a reason. You give up lots of control when you use them. In exchange, you get "crowd immunity" -- the hope that if there is a vulnerability, it will affect so many people that (1) you are not going to be the easiest target and (2) someone, somewhere will notice it.
Your build actions happen all in the docker images/ephemeral VMs. You use images directly distributed by the corresponding project, for example you may start directly from Canonical's Ubuntu image. The "runners" are provided by Github, and managed by Microsoft's security team as well. The only thing that you actually control is a 50-line YAML file in your git repo, and people will look at it any time they want to add a new feature.
Yes, the if someone hacks Microsoft's ops people, they can totally mess up my day. But would they? Every usage of zero-day carries some risk, so if attackers do get access to those systems, they'll much likely to go for some sort of high-value, easy-money target like cryptocurrency exchanges. Plus, I am pretty sure that Microsoft actually has solid security practices, like automatic deployments, 2FA everywhere, logging, auditing, etc... They are not going to have a file on CI/CD machine that is different from one in Git, like OP's system did!
So, you are not concerned about APTs at all. Well, nothing on the entire thread is relevant to you.
Um, no, that’s not what I said.
The APTs do not have magical powers, they buy from the same exploit market everyone has.
Let’s say my organization (which is not very well known) has an exploitable bug. What are the chances that someone will discover it? Pretty close to none, the hole can be there for many years waiting for APT to come and exploit it.
Now imagine Github runner or default Ubuntu image has an exploitable bug. What are the chances it will last long? Not very high. In a few months, someone will discover and either report or exploit it. Then it will be fixed and no longer helpful for APT threat actors.
Remember, the situation described in the post only occurred because they used binary images that only a few people could look at. Generating binary kernel on someone’s laptop is easy to subvert in undetectable way, but how do you subvert a Dockerfile stored in Git repo without it being obvious?