Heroku April 2022 Incident Review | Hacker News Reader

Heroku April 2022 Incident Review

blog.heroku.com

118 pointsglennericksen13 comments

> We began investigating how the threat actor gained initial access to the environment and determined it was obtained by leveraging a compromised token for a Heroku machine account. We determined that the unidentified threat actor gained access to the machine account from an archived private GitHub repository containing Heroku source code. We assessed that the threat actor accessed the repository via a third-party integration with that repository. We continue to work closely with our partners, but have been unable to definitively confirm the third-party integration that was the source of the attack

So they still don't know how it happened.

That's basically the only paragraph of new information in the whole... well, let's call it what it is, a press release basically.

It is a thing to know. That they had a token checked into source code that probably shouldn't have been, and that the attacker somehow got access to the source code in a private github repo.

* yeah, we still don't know how. Which is kind of important. But figuring out how they got access to a private github repo is to some extent back on github, at least potentially...

* it to some extent points back to github (fairly or unfairly); so how did they get access to the source code you were supposed to be protecting?

* it certainly reminds us the readers why we don't put secrets in source code repos. (Of course, they ultimately need to be stored somewhere, and that somewhere can always be breached. But having them in as few places as possible and places designated specifically for secrets, we can make sure that things like a Github OAuth token for a Github integration doesn't give access to your deploy secrets...)

* Which again makes me wonder... "via a third-party integration with that repository"... WAS it an integration that didn't even need source code read access, but had it due to Github's terribly non-specific integration auth permissions? I would love some more attention to that github problem as a result of this hullaballo. I kind of can't believe fixing integration permissions granularity has been such a low priority for a fairly well-resourced github. I guess it doesn't sell more accounts... it just loses you some once it results in a vulnerability, if that becomes known.

> is to some extent back on github

How is it on Github? We know that it was a third-party integration that compromised. Almost every single third-party integration needs the ability to read source code. The fault here lies on Heroku for storing secrets that allowed access to their main customer database in a source code repo that was accessible to a third-party provider, as well as with the third party provider (whatever it was) for allowing their implementation to be compromised. At that point it's game over—your main database should never be accessible from your source code alone.

It is strange to me that they're certain it was accessed through a third-party Github integration, but they don't know which integration it was specifically. That feels like a failure of logging on Github's part, without any additional information.

> How is it on Github?…

The whole sentence I wrote was: "But figuring out how they got access to a private github repo is to some extent back on github, at least potentially..."

> …That feels like a failure of logging on Github's part, without any additional information.

There you go, you answered your own question too.

You don't like granting report admin access to tokens that just need to read issues?

It's simple.

It's a little hard to parse parts of that paragraph, but it sounds like the repo (presumably hosted on GitHub) had access tokens granted to third party integrations (similar to Heroku being granted access to GitHub on behalf of their mutual users).

Assuming that's true, it should be trivial for GitHub to tell them which third party integration the token was associated with.

AIUI, the repo contained a single token that gave access to Heroku. Additionally, a bunch of third party tools had legitimate access to the repo. Any one of them could have been used to steal the token.

Is it known who at Github can access private customer repos?

I'd assume everyone who works there. They aren't encrypted or anything. If you have hardware access you have full github access.

But I can definitively say that any repo access is heavily logged. And this info is from like almost 10 years ago when I had first hand experience. I know they've upped their game in terms of repository access gating since then.

Maybe it was via Travis CI? https://blog.travis-ci.com/2022-04-17-securitybulletin

> Additionally, according to GitHub, the threat actor accessed and cloned private repositories stored in GitHub owned by a small number of our customers. When this was detected, we notified customers on April 15, 2022, revoked all existing tokens from the Heroku Dashboard GitHub integration, and prevented new OAuth tokens from being created.

Various customers received an email from Heroku on April 15 saying "We value transparency and wanted to notify you of an incident we're actively investigating that may lead to unauthorized access to your GitHub repositories connected to Heroku."

The way this incident review (to call it a post-mortem would be an insult to those who write good post-mortems) phrases things, customers have no way of knowing if that email meant they were one of those "small number of customers" or not. And what is a small number, anyways? Is 49% of customers a small number of customers? It's an absurd situation.

Side note on Heroku: they've been very aggressively upgrading old app dependencies this week (I've had ~9 apps go down for mandatory Postgres maintenance in the last 24h, with almost no notice for some of them). With how little information they've given regarding this incident, I can't help but wonder if it's related.

Reply on news.ycombinator.com