Comment by mwcampbell
5 years ago
I'm trying to find the lesson in here about how to prevent this kind of incident in the first place. The nearest I can find is: don't build any production binaries on your personal machine.
5 years ago
I'm trying to find the lesson in here about how to prevent this kind of incident in the first place. The nearest I can find is: don't build any production binaries on your personal machine.
Reproducible builds can go a long way, along with a diverse set of build servers which are automatically compared. Whether you use your personal machine or a CI system there's still the risk of it being compromised (though your personal machine is probably at a little more risk of that since personal machines tend to have a lot more software running on them than CI systems or production machines).
I'm paranoid, and I'd have considered the efforts described here to be pretty secure. I'll say the only counter to this grade of threat is constant monitoring, by a varied crew of attentive, inventive, and interested people. Even then, there's probably going to be a lot of luck needed.
Traffic analysis and monitoring will detect detect signs of intrusion almost in real time but also exfiltration. The network never lies.
The kind of eyes that can spot the hinky pattern while watching that monitor are the vital ingredient, and thats not something i can quantify. Or even articulate well.
> The network never lies.
Steganography begs to differ.
How much free entropy do you have on your network traffic?
EDIT: Corrected. Thanks cuu508.
1 reply →
One sensible mitigation to this grade of threat; avoid running Windows, even as a VM host as the dev did. It's a dumpster fire.
I think you may have misinterpreted that part of the post - my understanding is that the Linux laptop that was being used was compromised, and there was a 3 month gap when that developer switched to a Windows machine before that became compromised too. Specifically it would be fascinating to learn whether the Windows host was compromised or if it was only the Linux VM.
4 replies →
I would go further and say:
"Developer systems are often the weakest link."
(Assuming that the system on itself is designed with security in mind.)
The reason is manifold but include:
- attacks against developer systems are often not or less considered in security planing
- many of the technique you can use to harden a server conflict with development workflows
- there are a lot of tools you likely run on dev systems which add a large (supply chain) attack surface (you can avoid this by allways running everything in a container, including you language server/core of your ides auto completion features).
Some examples:
- docker groub member having pseudo root access
- dev user has sudo rights so key logger can gain root access
- build scripts of more or less any build tool (e.g. npm, maven plugins, etc.)
- locking down code execution on writable hard drives not feasible (or bypassed by python,node,java,bash).
- various selinux options messing up dev or debug tools
- various kernel hardening flags preventing certain debugging tools/approaches
- preventing LD_PRELOAD braking applications and/or test suites
...
I think a big difference between build machines and dev machines, at least in principle, is that you can lock down the network access of the build machine, whereas developers are going to want to access arbitrary sites on the internet.
A build machine may need to download software dependencies, but ideally those would come from an internal mirror/cache of packages, which should be not just more secure but also quicker and more resilient to network failures.
Interestingly, this is water on mills we are currently thinking about. We're in the process of scaling up security and compliance procedures, so we have a lot of things on the table, like segregation of duties, privileged access workstations, build and approval processes.
Interestingly, the way with the least overall headaches is to fully de-privilege all systems humans have access to during regular, non-emergency situations. One of those principles would be that software compiled on a workstation automatically disqualifies from deployment, and no human should even be able to deploy something into a repository the infra can deploy from.
Maybe I should even push container-based builds further and put up a possible project to just destroy and rebuild CI workers every 24 hours. But that will make a lot of build engineers sad.
Do note that "least headaches" does not mean "easy".
This is why I always insist on branches being protected at the VCS server level so that no code can sneak in without other's approval - the idea is that even if your machine is compromised, the worst it can do is commit malicious code to a branch and open a PR where it'll get caught during code review, as opposed to sneakily (force?) pushing itself to master.
In this case no CI was involved so that wouldn't have helped.
(The CI was not compromised but a dev laptop which was used to manually build+deploy the kernel, without any CI involved).
Through generally I agree with you.
If you use cloud services that offer automated builds you can push the trust onto the provider by building things in a standard (docker/ami) image with scripts in the same repository as the code, cloned directly to the build environment.
If you roll your own build environment then automate the build process for it and recreate it from scratch fairly often. Reinstall the OS from a trusted image, only install the build tools, generate new ssh keys that only belong to the build environment each time, and if the build is automated enough just delete the ssh keys after it's running. Rebuild it again if you need access for some reason. Don't run anything but the builds on the build machines to reduce the attack surface, and make it as self contained as possible, e.g. pull from git, build, sign, upload to a repository. The repository should only have write access from the build server. Verify signatures before installing/running binaries.
> If you use cloud services that offer automated builds you can push the trust onto the provider by building things in a standard (docker/ami) image with scripts in the same repository as the code, cloned directly to the build environment.
And I guess, for those super-critical builds, don't rely on anything but the distro repos or upstream downloads for tooling?
Because if you deploy your own build tools from your own infra, you are at risk to taint the chain of trust with binaries from your own tainted infra again. I'm aware of the trusting trust issue, but compromising the signed gcc copy in debians repositories would be much harder than some copy of a proprietary compiler in my own (possibly compromised) binary repository.
> And I guess, for those super-critical builds, don't rely on anything but the distro repos or upstream downloads for tooling?
You can build more tooling by building it in the trusted build environment using trusted tools. Not everything has to be a distro package, but the provenance of each binary needs to be verifiable. That can include building your own custom tools from a particular commit hash that you trust.
Did you mean read access from the build server? I’m confused.
I believe the commenter meant that only the build server should be able to write to the build artifact repository, so ”write access from the build server” would be correct.
1 reply →
Hum... On what machine do you build them?
That can not be the right lesson, because there's no inherent reason "personal machine" is any less safe than "building cluster" or whatever you have around. Yes, on practice it often is less secure to a degree, so it's not a useless rule, but it's not a solution either.
If it's solved some way, it's by reproducible builds and automatic binary verification. People are doing a lot of work on the first, but I think we'll need both.
> there's no inherent reason "personal machine" is any less safe than "building cluster" or whatever you have around
Sure there is! I browse internet a lot on my dev machines, and this exposes me to bugs in browsers and document viewers. And if I do get compromised, my desktop is so complex and runs so many services the compromise is unlikely to be detected. So all attacker needs is one zero day, once.
Compare this to a CI with infra-as-a-code, like Github Actions. If the build process gets compromised, it only matters until the next re-build. Even if you get a supply chain attack once (for example), if this is discovered all your footholds disappear! And even if you got the developers' keys, it is not easy to persist -- you have to make commits and those can be noticed and undone.
(Of course if your "building cluster" is a bunch of traditional machines which are never reformatted and which many developers have root access to, then they are not that much more secure. But you don't have to do it that way.)
You rebuild your build cluster with what image? Where do the binaries there come from? And what machine rebuilds the machines.
Securing the machines themselves is a process of adding up always decreasing marginal gains until you say "enough", but the asymptote is never towards a fully secure cluster. That ceiling on how secure you can get is clearly suboptimal.
Besides, the ops people's personal machines have a bunch of high access permissions that can permanently destroy any security you can invent. That isn't any less true if your ops people work for Microsoft instead of you.
3 replies →
Use a PaaS like Heroku or Google App Engine, with builds deployed from CI. All the infrastructure-level attack surface is defended by professionals who at least have a fighting chance.
I feel reasonably competent at defending my code from attackers. The stuff that runs underneath it, no way.
The lesson is in the essay from James Mickens, above.
But isn't that just defeatist? Can't we continue to ratchet up our defenses?
If you double your defences, you double the cost but advanced attackers still get what they want. "Ratchet up defences" does't mean simply doing things a bit more correctly, it requires you to hire many expensive people to do lots of things that you didn't do before. This article is a good example - the company as described seems to have a very good (and expensive) level of security already, the vast majority of companies are much less secure, and it still wasn't sufficient.
And if you increase your defences so much that you're actually somewhat protected from an advanced attacker, you're very, very far on the security vs usability tradeoff, to get there is an organization-wide effort that (unlike simple security basics/best practices) makes doing things more difficult and slows down your business. You do it only if you really have to, which is not the case for most organizations - as we can see from major breaches e.g. SolarWinds, the actual consequences of getting your systems owned are not that large, companies get some bad PR and some costs, but it seems that prevention would cost more and still would be likely to fail against a sufficiently determined attacker.
1 reply →
in solarwind hack their cicd systems were hacked, don't see a lesson here
Build everything on a secured CI/CD system, keep things patched, monitor traffic egress especially with PII, manual review of code changes, especially for sensitive things