Codecov Security Event and HashiCorp GPG Key Exposure

One thing to consider is that manual signing, even on an air-gapped machine, doesn't buy more security if your entire CI system itself can be compromised, unless the person responsible for manual signing is really able to verify that what was built matches the code. The only feasible way to do that is with reproducible builds on randomly selected servers that are then compared to each other, but even that assumes an attacker can't compromise every build server. The only way to do that is to have at least one build server be disconnected, but then you need all dependencies available locally.

Either way, there is tremendous complexity in trying to achieve this. Realistically, you should take artifact signing to mean what you're installing is what the vendor actually built and not something shipped by a man in the middle. It does nothing to protect you from the vendor itself being compromised.

For questions like "why is the control system of that bridge on the Internet," consider the case a few years back when I first moved to Dallas that someone managed to compromise the city's tornado warning system because it was entirely controlled by unencrypted radio signals. The system predated the Internet. Somebody managed to set off alarms citywide for nearly an entire day by just pointing some handheld transmitter at a tower.

Unfortunately, the security posture of any infrastructure controlled by effectively anyone other than the military is close to no security at all in many cases, especially older infrastructure.

I doubt they could verify it otherwise they could have simply stated they have verification the key wasn't compromised. Instead they said an "investigation has not revealed evidence" which can mean anything. Just looking at a bunch of logs (or even if you don't have any logs) you can claim the investigation "has not revealed evidence".