What happens when you leak AWS credentials and how AWS minimizes the damage | Hacker News Reader

What happens when you leak AWS credentials and how AWS minimizes the damage

xebia.com

201 pointstiborhercz66 comments

I set up a project[1] to automatically leak AWS secrets published to the Python package index, which then triggers the secret scanning process and quarantines the keys[2]

1. https://github.com/pypi-data/pypi-aws-secrets

2. https://github.com/pypi-data/pypi-aws-secrets/blob/main/keys...

Chaotic Good?

It is really silly that the only easy way to report a leaked secret from most services is to push it to GitHub. I don't understand how the industry failed to standardize any secret-reporting scheme.

How would this work in practice? In order to effectively filter for tokens you need a set of pretty specific regular expressions.

These specific expressions may contain non-public information about the structure of the tokens.

But imagine we live in a world where companies are OK with exposing this information publicly and investing time in keeping the expressions up to date.

Now what? How do you report it to services whilst stopping abuse? Like, if the code isn’t public then all this is an endpoint to let anyone to disable another users (or companies) key without any explanation or recourse.

Pushing to GitHub is actually quite an elegant solution to these issues I think, even if I do think it is a bit too centralized.

I mean yes — that’s the point. If untrusted parties have access to your keys, it’s already game over. You’ve lost. Disabling them is the nicest thing an attacker can do for you.

Not really: it depends on the permissions assigned to the keys.

I wouldn’t like to wake up to an email that says “your key has been disabled because someone anonymously reported is as leaked, sorry if this has broken your entire system”.

What do you do with this, outside of obviously quarantining and/or disabling the key? How was it leaked? What’s the context?

If rogue party has access to the key, they can do a lot more damage.

Indeed, so you’re building all this tooling and complexity and introducing more issues for the very small intersection of people that:

1. Are not malicious

2. Have access to a key

3. Are unable or unwilling to commit it to GitHub

It would be great if this stuff was public and available without a central authority. But after working on it for a while it seems like a fairly good compromise.

There’s already an entire pipeline that handles a key being compromised when it is found on GitHub. All the “tooling and complexity” you need is a simple HTML form to ask for a key and where you found it, and some server-side code to trigger the same pipeline when somebody submits the form.

There are three issues with the use of GitHub here:

1. Not everybody knows that AWS will invalidate tokens committed into a public GitHub repository.

2. There is a window (67 seconds according to OP) in which the compromised token is public but working. For the “small intersection of people”, you could bring it down to 0.

3. GitHub protects GitHub keys, and apparently AWS keys, but does it protect Azure keys? Or GCP keys?

GitHub secret scanning is a product they allow partners to take advantage of: https://docs.github.com/en/code-security/secret-scanning/abo...

https://docs.github.com/en/code-security/secret-scanning/sec...

Not necessarily people but systems. Your code hosting platform, your mailing-list host, your chat app, ... are all examples of systems who are hopefully not malicious and could easily add this auto-revoke-publicly-leaked-secret feature, if it was a simple consistent scheme like an HTTP DELETE to a URL.

I'd like a standardized token syntax that contains a domain (e.g. `secret:example.com/token`) Then you could fetch an url like `https://example.com/.well-known/secret-leak` which returns machine readable information about how to report a leaked token.

I keep arguing for this too! [1][2][3]

A few cycles ago [4] sudhirj pointed out that you could just use an actual URL as your secret key and have that be the URL you visit to revoke it, and I think that is genius.

[1] https://news.ycombinator.com/item?id=34009442

[2] https://news.ycombinator.com/item?id=31335081

[3] https://news.ycombinator.com/item?id=25979908

[4] https://news.ycombinator.com/item?id=28299624

This is a misunderstanding of how standards come about. Standards don't come about because someone theorizes pushing the envelope and then everyone gets together to form a convention to decide the best way to push the envelope.

Standards describe existing behaviour of value that benefits from a single interface. For that, the behaviour has to be first demonstrated. That's how the envelope is pushed. You make an implementation, then you show how that solves something worthwhile, and then if there are multiple implementations that could benefit from a single interface, you do that.

Totally agree, and I do hope that at some point this will be standardised.

Ideally you’d have an endpoint that returns a set of patterns to check for and a way to submit a set of tokens for checking, along with some kind of context. It would probably be something like what GitHub requires[1].

Unfortunately that has somewhat limited value when GitHub can do most of the legwork for you, and as the de-facto place to publish code the majority of “accidental secrets exposed in code” are leaked there.

1. https://docs.github.com/en/code-security/secret-scanning/sec...

At the very least I would think GitLab might be interested in providing a similar feature.

That's right, the behavior exist, on GitHub. Multiple services, including open-source ones like PyPI, integrated with it because they find it worthwhile. Yet it remains impossible for anyone to use it except through GitHub.

Really interesting experiment!

To answer "how do the bad guys find them so quickly" the answer is "judicious queries to the GitHub API."

My research group describes how this works, and what we could see with it, in this paper: https://bradreaves.net/mmr19.pdf

We're currently working on follow up work to make secret management better (with NSF funding[1]), and we're always looking for research collaborators[2].

[1] https://nsf.gov/awardsearch/showAward?AWD_ID=2055554&Histori...

[2] https://bradreaves.net/collaborate/

Tailing this presumably: https://api.github.com/events

Apparently not; there’s reportedly a 5 minute delay on that feed.

A few years ago this wasn't in place. I goofed and leaked a credential via Github, and woke up to $50k and climbing in AWS fees (Monero miners). AWS worked with us, and once we proved we had adequately locked our account down and added additional fail-safes, they took it off of our bill.

> In the email they also give guidance on how to secure the account. They do this in four steps: 1. Rotate and delete the exposed AWS Access Key...

I work at a very small shop (at a non-profit institution).

In the past, when we've accidentally leaked a credential, AWS has required us to rotate all of our AWS Access Keys in the account, not just the exposed one.

Which I know ideally rotation would be easy/automated and this wouldn't matter, but at the time it was disruptive for us. And a bit frustrating when we had intentionally created separate AWS credentials with least privs needed for the service, etc. But thinking on it, I guess any given credential might give permissions to create other credentials or other devious things, rotating all of them might be necessary to be safe.

It's been a couple years since we had a leaked credential now (hooray). I wonder if the policy has changed, or if I misunderstood, or if in fact you still need to rotate all credentials in the account, maybe OP just only had one credential in the account they were intentionally doing insecure things so the compromised key was in fact all keys. (I don't specifically remember the `AWSCompromisedKeyQuarantineV2` policy, but probably am just not remembering it, I had other things to worry about at the times!)

Can also avoid having any static credentials at all if possible.

This used to not be possible. AWS only added support for OIDC Credentials in 2014, which is 10 years ago but well after AWS launched. Convenient support in client libraries was only added in 2019[1][2].

[1] https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-f... [2] https://github.com/aws/aws-cli/releases/tag/1.16.232

   12:33:12 – Pushed the credentials to GitHub

   12:34:19 – The AWSCompromisedKeyQuarantineV2 policy is attached to the IAM user test-user by AWS

Ah, so my exploit must detect and execute within about a minute. Seems doable.

This is the equivalent of the crime movie where they start the heist and then start the stopwatch.

“Ok, secret is pushed, we have 77 seconds until Amazon shuts us off. Let’s go!!!!”

Where are you getting the additional 10 seconds to pull off this heist?

Ha, the magic of my bad arithmetic. It should be 67 seconds.

Harold Faltermeyer's tune kicks into high gear: https://www.youtube.com/watch?v=XfYgW3Uj9g4&t=107s

Doable for an automated system. Anything that requires human intervention would be way harder.

Plus the time it takes for IAM eventual consistency to catch up. These still went through, there was just nothing interesting available:

    12:34:32 – Various List and Describe calls are made using the leaked credentials

"What to Do If You Inadvertently Expose an AWS Access Key" - https://aws.amazon.com/blogs/security/what-to-do-if-you-inad...

"What do I do if I notice unauthorized activity in my AWS account?" - https://repost.aws/knowledge-center/potential-account-compro...

Ever wondered what would happen when you leak AWS credentials to GitHub? I was, so I leaked AWS credentials to GitHub to see what would happen. Both AWS and malicious actors respond very quickly and start taking actions on these leaked credentials.

"The Anatomy of an AWS Key Leak to a Public Code Repository" (2019) - https://blog.technodrone.cloud/2019/03/the-anatomy-of-aws-ke...

Has anyone got a script or git hook that will suggest you not to push anything that looks like a credential? Could be really useful.

It would need to identify strings that look like a key/password, and I guess also it would need an override, perhaps a comment like "// Demo password".

GitHub has some protections that you can enable to help prevent this.

https://github.blog/2022-04-04-push-protection-github-advanc...

Note: paid add-on, and I believe it requires an enterprise subscription to start with.

There exists some open source tooling in the same space.

There are several tools which can be used with pre-commit, for example trufflehog.

I've used sync-secrets before, which has worked well- https://github.com/Yelp/detect-secrets

I use one a git pre-commit hook that according to my notes i copied from here (at some point in the past, it may have changed since then):

https://gist.github.com/czardoz/b8bb58ad10f4063209bd

Then adapted with a better regex for secret keys I got from here: https://github.com/awslabs/git-secrets/blob/80230afa8c8bdeac...

(I think maybe I was getting false positives before)

The excellent pre-commit framework (https://pre-commit.com/) has a hook for that in its official hook collection: https://github.com/pre-commit/pre-commit-hooks#detect-aws-cr...

I think it will be great if AWS allowed us to use the AWS keys to be used from certain IPs only. It will add additional layer of security which can help to prevent a lot of misuse. Just like AWS best practices of putting DB instances in a private VPC can prevent a lot of attack vectors, ability to use AWS Keys only from certain IPs or IP ranges, can also prevent from misuse, even if one accidentally leaks certain keys.

This looks to be possible via this guide https://repost.aws/knowledge-center/iam-restrict-calls-ip-ad...

This example is for a role but for IAM users it works the same way.

I don't think AWS wants to imply in any way that you need to be on a VPN to interact with AWS resources. It's not the direction they are going (Zero trust / RBA https://aws.amazon.com/security/zero-trust/).

Zero trust doesn’t mean no VPNs, it means not just VPNs. In other words you can put your control plane behind ACLs (and you should!) but you should still require strong auth from the private network.

It's already possible, but on a iam action level.

Source IP is an IAM global condition so it's available for all requests.

https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_p...

Haven't tried it, but should be able to use service control policies to do this broadly.

https://docs.aws.amazon.com/organizations/latest/userguide/o...

Can definitely do it at the permission policy level as shown in the links others provided.

Basically, rotate as soon as you can* and start looking through your AWS logs and setting to see if any services you don't recognize have been spun up. Is you think you have been attacked or see stuff you did not spin up, contact AWS support ASAP!

*Do NOT just revoke keys if it is in a production system where other people are working or are depending on. Talk to your team and figure out what the remediation process is internally and follow that! If you are working by yourself and no one is relying on services this key is associated with, then yeah, just revoke and replace ASAP.

Here are a couple blogs that cover what you are looking for as well https://blog.gitguardian.com/leaking-secrets-on-github-what-... https://blog.gitguardian.com/thinking-like-a-hacker-aws-keys...

Please be aware of leaking AWS credentials in any log output, especially Jenkins console output. That's usually protected by some layer of auth, but if I get some chump's account info, and something is leaking AWS creds to Jenkins console output, it's probably gonna end up with full account take-over.

Discord does a similar thing, if a bot toke is discovered online it automatically gets disabled and you get an email.

OB: There is never a reason to have access keys anywhere in your source code or a config file in your source control directory.

All of the AWS SDKs will get your credentials from

- hardcoded credentials in your code -> don’t do this

- environment variables - you get this when running locally from your control tower interface that most companies access via SSO via AD/Okta etc

- your global configuration file.

When running on AWS, your credentials are retrieved via the attached IAM Role for Lambda, ECS, EC2, etc.

I have done one third party integration where I used Azure DevOps to integrate with AWS using the AWS tools for Azure DevOps. Even then you store your credentials in an Azure vault separate from your source code.

I can understand how AWS can react quickly using the secrets scanning service but how do malicious actors do it without having access to that service?

Luck, I guess?

Maybe they have a bot watching some "latest thing on github" API, that scans everything coming in for credentials.

there's a public firehose feed of everything on github

https://api.github.com/events

(not sure why really)

It says in the docs this API is delayed 5 minutes though. So no idea how they did things this quickly unless you can add a webhook to every repository somehow as it is created?

It's probably just polling and luck to be honest.

Maybe the 5 minutes delay is so partners like AWS get a chance to scan first.

https://docs.github.com/en/code-security/secret-scanning/abo...

This is one of those things that I was somewhat aware existed since we had seen various reports of it, but I was never fully aware of the speed (and obviously automated) of it.

Is this actually documented by aws anywhere? I assume other cloud providers must be doing something similar.

OpenAi does the same. Can this somehow used to quarantine other credentials? F.e. is it checking the key and the secret? Or is exposing the key sufficient?

It’s interesting how well cryptos have adapted to monetizing stolen aws credentials.

I really liked this blog, thanks for sharing!

Interesting blog, thanks for sharing!

It's not clear to me that we have proof that it was a malicious actor. Was it perhaps AWS automation testing out the validity of the credentials before applying the leaked credential policy?

The AWS process is hopefully "does this match a secret we know? invalidate!" and nothing more. Relying on "can this key do x?" would be a problem, because it might be able to do x in the future, and it doesn't appear they're testing all possible capabilities either.

Reply on news.ycombinator.com