I'm trying to find the lesson in here about how to prevent this kind of incident in the first place. The nearest I can find is: don't build any production binaries on your personal machine.
Steganography begs to differ.
How much free entropy do you have on your network traffic?
EDIT: Corrected. Thanks cuu508.
How the attackers were able to gain access again after the developer used a VM in Windows? My guesses:
- The developer machine was compromised in a deeper level (rootkit?)
- The developer installs a particular application in each Linux box
- There is a bug in an upstream distro
Unlikely that would not have taken 3 month.
> The developer installs a particular application in each Linux box
Possible, but also unlikely, as long as the vm wasn't used for other things this also wouldn't have taken 3 month.
> The developer installs a particular application in each Linux box
There probably is, but it probably has nothing to do with this exploit. For the same reasons as mentioned above.
My guess is that it was a targeted attack against that developer and there is a good chance the first attack and the second attack used different attack vectors hence the 3 month gap.
"Developer systems are often the weakest link."
(Assuming that the system on itself is designed with security in mind.)
The reason is manifold but include:
- attacks against developer systems are often not or less considered in security planing
- many of the technique you can use to harden a server conflict with development workflows
- there are a lot of tools you likely run on dev systems which add a large (supply chain) attack surface (you can avoid this by allways running everything in a container, including you language server/core of your ides auto completion features).
Some examples:
- docker groub member having pseudo root access
- dev user has sudo rights so key logger can gain root access
- build scripts of more or less any build tool (e.g. npm, maven plugins, etc.)
- locking down code execution on writable hard drives not feasible (or bypassed by python,node,java,bash).
- various selinux options messing up dev or debug tools
- various kernel hardening flags preventing certain debugging tools/approaches
- preventing LD_PRELOAD braking applications and/or test suites
...
A build machine may need to download software dependencies, but ideally those would come from an internal mirror/cache of packages, which should be not just more secure but also quicker and more resilient to network failures.
Interestingly, the way with the least overall headaches is to fully de-privilege all systems humans have access to during regular, non-emergency situations. One of those principles would be that software compiled on a workstation automatically disqualifies from deployment, and no human should even be able to deploy something into a repository the infra can deploy from.
Maybe I should even push container-based builds further and put up a possible project to just destroy and rebuild CI workers every 24 hours. But that will make a lot of build engineers sad.
Do note that "least headaches" does not mean "easy".
(The CI was not compromised but a dev laptop which was used to manually build+deploy the kernel, without any CI involved).
Through generally I agree with you.
If you roll your own build environment then automate the build process for it and recreate it from scratch fairly often. Reinstall the OS from a trusted image, only install the build tools, generate new ssh keys that only belong to the build environment each time, and if the build is automated enough just delete the ssh keys after it's running. Rebuild it again if you need access for some reason. Don't run anything but the builds on the build machines to reduce the attack surface, and make it as self contained as possible, e.g. pull from git, build, sign, upload to a repository. The repository should only have write access from the build server. Verify signatures before installing/running binaries.
And I guess, for those super-critical builds, don't rely on anything but the distro repos or upstream downloads for tooling?
Because if you deploy your own build tools from your own infra, you are at risk to taint the chain of trust with binaries from your own tainted infra again. I'm aware of the trusting trust issue, but compromising the signed gcc copy in debians repositories would be much harder than some copy of a proprietary compiler in my own (possibly compromised) binary repository.
You can build more tooling by building it in the trusted build environment using trusted tools. Not everything has to be a distro package, but the provenance of each binary needs to be verifiable. That can include building your own custom tools from a particular commit hash that you trust.
That can not be the right lesson, because there's no inherent reason "personal machine" is any less safe than "building cluster" or whatever you have around. Yes, on practice it often is less secure to a degree, so it's not a useless rule, but it's not a solution either.
If it's solved some way, it's by reproducible builds and automatic binary verification. People are doing a lot of work on the first, but I think we'll need both.
Sure there is! I browse internet a lot on my dev machines, and this exposes me to bugs in browsers and document viewers. And if I do get compromised, my desktop is so complex and runs so many services the compromise is unlikely to be detected. So all attacker needs is one zero day, once.
Compare this to a CI with infra-as-a-code, like Github Actions. If the build process gets compromised, it only matters until the next re-build. Even if you get a supply chain attack once (for example), if this is discovered all your footholds disappear! And even if you got the developers' keys, it is not easy to persist -- you have to make commits and those can be noticed and undone.
(Of course if your "building cluster" is a bunch of traditional machines which are never reformatted and which many developers have root access to, then they are not that much more secure. But you don't have to do it that way.)
Securing the machines themselves is a process of adding up always decreasing marginal gains until you say "enough", but the asymptote is never towards a fully secure cluster. That ceiling on how secure you can get is clearly suboptimal.
Besides, the ops people's personal machines have a bunch of high access permissions that can permanently destroy any security you can invent. That isn't any less true if your ops people work for Microsoft instead of you.
Your build actions happen all in the docker images/ephemeral VMs. You use images directly distributed by the corresponding project, for example you may start directly from Canonical's Ubuntu image. The "runners" are provided by Github, and managed by Microsoft's security team as well. The only thing that you actually control is a 50-line YAML file in your git repo, and people will look at it any time they want to add a new feature.
Yes, the if someone hacks Microsoft's ops people, they can totally mess up my day. But would they? Every usage of zero-day carries some risk, so if attackers do get access to those systems, they'll much likely to go for some sort of high-value, easy-money target like cryptocurrency exchanges. Plus, I am pretty sure that Microsoft actually has solid security practices, like automatic deployments, 2FA everywhere, logging, auditing, etc... They are not going to have a file on CI/CD machine that is different from one in Git, like OP's system did!
The APTs do not have magical powers, they buy from the same exploit market everyone has.
Let’s say my organization (which is not very well known) has an exploitable bug. What are the chances that someone will discover it? Pretty close to none, the hole can be there for many years waiting for APT to come and exploit it.
Now imagine Github runner or default Ubuntu image has an exploitable bug. What are the chances it will last long? Not very high. In a few months, someone will discover and either report or exploit it. Then it will be fixed and no longer helpful for APT threat actors.
Remember, the situation described in the post only occurred because they used binary images that only a few people could look at. Generating binary kernel on someone’s laptop is easy to subvert in undetectable way, but how do you subvert a Dockerfile stored in Git repo without it being obvious?
I feel reasonably competent at defending my code from attackers. The stuff that runs underneath it, no way.
And if you increase your defences so much that you're actually somewhat protected from an advanced attacker, you're very, very far on the security vs usability tradeoff, to get there is an organization-wide effort that (unlike simple security basics/best practices) makes doing things more difficult and slows down your business. You do it only if you really have to, which is not the case for most organizations - as we can see from major breaches e.g. SolarWinds, the actual consequences of getting your systems owned are not that large, companies get some bad PR and some costs, but it seems that prevention would cost more and still would be likely to fail against a sufficiently determined attacker.
Do you think having a "kernel guy" building the kernel on his personal laptop before deploying it to production is a very good level of security?