Comment by jrockway | Hacker News Reader

Comment by jrockway | Hacker News Reader

Parent Full thread

The other thing that is interesting about Codecov is how they distribute their analysis/uploader. You download a fresh uploader for every CI run, even if you're using your CI platform's native integration (i.e. the CircleCI "orb"). I get why they do this -- if they instructed people to download a binary and store it, customers would do that exactly once and never update it, causing them to have to support old versions of their API indefinitely (or annoy their customers greatly). So basically, we took a shortcut that triggers everyone's security spidey sense, and exactly as predicted, we all got burned.

(I will point out that their uploader is literally a curl | bash, but that isn't really the problem here. Installing a Debian package or downloading a statically linked Go binary would be just as risky. Actually, it would have been harder to find the underlying problem.)

All in all I'm pretty disappointed. We trusted Codecov with a shortcut that makes it easy for them to deliver a better product. They didn't have any security in place to ensure that what was checked into source control was what customers downloaded, and now we're all hacked.

You can use their bash uploaded, but you can also use language specific ones.

The Python one, for example, is downloaded via PyPI, and you’ll usually have a pinned version with the hash of it. Any tampering would make pipelines fail. If a new release were pushed by rogue actors, someone would notice an unplanned version bump.

FYI their support actively asks you to move to the bash uploader from the Python one, and the Python one had other compatability issues.

Here's a snippet from a support thead with them last year:

>One thing I noticed is that you are using the python uploader to upload coverage reports to Codecov. We do not support this uploader as a first-party tool, but we do support the bash uploader (https://docs.codecov.io/v4.3.0/docs/about-the-codecov-bash-u...) with full-time staff. We want to make sure this issue isn't related to something in a third party uploader before troubleshooting further. > Would you be able to switch to using this uploader to see if it's related?

I'd stopped using codecov last year for pricing reasons, when they moved to per-user pricing.

Debian packages can be signed, so depending on how the signing keys are managed - it might have been much easier to find the underlying problem.

True! I think since in the CI case you're bootstrapping a fresh machine for every build, you'd have to apt-key add their key as part of installing the package, so an attacker would just make a new one of those. I bet the compromise duration would have been shorter, though; there's got to be one engineer that installs the Debian package on their workstation and would be confused when their release system's signature stopped working.

(And honestly, an attacker that can build a Debian package and update the public key for apt-key is probably harder to find than one that can edit a cleartext bash script. Security through hoping your adversaries can't perform tedious devops operations is no security at all, as they say, but in the real world it probably helps a tiny bit.)

I'm not above throwing public keys in the git repo for use in CI/CD jobs.

They offer, but you don't have to. Only add the uploader that passed your review and use that, not just anything from the interwebs.

Reply on news.ycombinator.com