Hacker News Reader

Comment by TobyTheDog123 | Hacker News Reader

I honestly can't think of a worse result of a hack of a CI/CD service than source code of companies being stolen. In my mind, this is akin to the Okta breach a while back, a ton of companies being hit hard through no fault of their own all at once.

I can appreciate the want to diversify services so that secrets/env are separate from code, but I think I would honestly trust the behemoth that is Github with both.

That being said, my company still uses Okta, so freebies and mulligans are certainly still tolerated when it comes to data breaches.

View on HN

Securing your supply chain should be one of the first steps to trying to secure your company.

I mean, it was preventable, you just had to not use the massive single point of failure that is a cloud CI app. But everything is "so much easier" and "so much cheaper" with the cloud I guess!

Having experienced cloud CI on several platforms as well as in-house installs of both Travis and Jenkins, I think the ratio of downtime between cloud and local makes the cloud a far, far better pick.

That said, I am currently very glad I'm not running anything on circle ci...

Hindsight is 20/20 for sure, but for this use case confidentiality seems more important than availability.It is something of an impossible choice, but I've come to think that every service big or small gets hacked sooner or later. I'm thinking being a smaller target is better in that case.

But a smaller target might mean less funds for security, as it's basically a constant overhead.

I’m fairly confident that in the same breach scenario (laptop with admin permissions taken over), most small orgs would fare worse. The CI system would likely be behind a VPN, but the laptop would likely have those credentials, so it would not stop an attacker.

A small, 20 person org has maybe 2 people assigned to ops, so monitoring and breach detection is likely worse.

Now, a small org may be a less attractive target and some orgs can have top notch security people, but on average, the trade-off is likely not in favor of hosting your own.

I had the opposite experience with a large UK corp when moving from self-hosted Gitlab to gitlab.com.

Runners were still self hosted, but if the thing controlling them is just giving 500s all day and you’ve no influence on fixing it, then your jobs aren’t being run and your developers are sitting somewhat idle.

Github Actions has been better, but not perfect… but you can still fully self-host this if you think it’s worth it !

Yes because every company that is not using the cloud for CI/CD have much better security.

I’ve seen internal hosted Jenkins servers give up the ghost more times than I care to mention.

The difference, hopefully, is that private Jenkins installation is behind corporate network. So for attackers to reach it, either they need to breach the network first or it’s an internal employee.

Personally i’m a big proponent of both cloud services and privately managed services. CI is one of those i think are better kept private due to its sensitivity.

You really haven’t seen all of the reports of internal git repos getting breached?

> So for attackers to reach it, either they need to breach the network first or it’s an internal employee

So for instance an employee getting their laptop infected with malware, or through phishing, or through one of the many vulnerabilities discovered regularly on enterprise VPN software?

There is a reason that many organisations are going away with VPNs and "corporate networks" all together - it gives a false sense of security that stuff behind it is protected by the VPN and leads to poor security practices inside like obsolete Jenkins installs.

Disclaimer: I work at a company that sells Zero Trust as a concept and associated software, but I've had this opinion since before joining (I've seen enough of 'there's a VPN and then lots of apps/services/servers with very poor auth practices because it's behind the VPN, why bother?')

As someone from the security domain you should know there is not one single tool/service that will solve all your security challenges. I did not claim VPN will.

Your example of employee laptop getting infected with malware should be remediated by proper device management. Which should include policies like forced updates, prevents software install, uses corporate firewall, up to date antivirus, etc.

Since you mentioned you work for a company that sells zero trust, your response sounds like your solution would replace VPN. Don’t get me wrong, but feels like you are wearing the sales hat now.

Yes, old fashion VPN appliances will be probably seize to exist. Replaced by modern equivalent like tailscale. Layered security is the answer.

I work at BigTech. We have been moving over the past two years not to require VPN access for anything and having granular access via MFA.

From what I hear, Google is moving toward the zero trust direction too.

>From what I hear, Google is moving toward the zero trust direction too

Google were the ones who started this trend of zero trust

But if you look at the recent hacks of auto companies, it was all driven by bad web apps, badly implemented SSO, bad REST APIs etc and they could get control of the cars because it was all public on the internet. A properly implemented VPN would have killed their ability to do those hacks dead. A VPN concentrates an encryption and auth system in one place, which is hopefully somewhat competent. The zero trust concept assumes every app is a web app, and every web app is secure enough to be exposed to anyone. Big assumptions to make.

I've never seen a Jenkins instance that did not need to be open to the public internet in order to support GitHub webhooks. Granted they were using IP based restrictions but it's still not behind a corporate network by any stretch in that setup.

Though I believe the almighty cloud is overrated, not all cloud providers are equal. As much as I dislike Google, they would not have messed up this badly. Ditto Microsoft with GitHub.

I think you’re over-estimating the amount of resources and expertise maintaining your own CI infrastructure requires.

The places I typically work for don’t have the operations capacity necessary. Nor do the small development teams have anyone other than me familiar enough with system administration to set things up in a reasonable amount of time.

A lot of junior developers have little to no Linux server experience. And I don’t really count following a step-by-step tutorial as experience. I have that experience and server problems can easily eat days of time the team can’t afford. :(

I think you meant that they are under-estimating

Reply on news.ycombinator.com