SolarWinds exposed FTP credentials in Public Github Repo: US Government Breach
savebreach.com
savebreach.com
Just anecdotally I'm looking at my own release infrastructure and it would take a bit more work to get compromised binaries to our customers than a leaked secret in a public git repo - and if it was reported that we leaked some key or password that gave the keys of the kingdom away, it would be trivial to change it and verify that we didn't push malware.
This article can explain how malware made it into a download server, maybe. But does it explain the fact the malware was signed? Maybe, but that suggests a bigger fuck up than leaking a password.
But I could access literally everything related to that organization and their clients from that single Github leak (of Jira creds).
https://news.ycombinator.com/item?id=10754194
https://web.archive.org/web/20151217205414/http://www.exfilt...
There are already comments here from users who presumably haven't read the article and are parroting this as fact.
The credentials by themselves don’t give execute privileges, or persistence, or in fact any access to non-SolarWinds systems. So both can be true.
I don't see the need for this editorialised submission title.
To start, I want to clear a potential confusion in my previous comment. By "technical wizardry" I meant "technical hacking wizardry," e.g. buffer overflows, ROP, etc.
That said, I think Stuxnet and this hack may actually both be characterized by some of the same kind of sophistication I was referring to. IIRC, one of the biggest unique things about Stuxnet was the application of professional software engineering practices to malware. It wasn't a quick and dirty thing that one or two guys whipped up, it was a project, developed professionally over time by a team, to meet a set of requirements and allow code-reuse, etc. Not so much hacking wizardry but an application of government-scale money and personnel.
Yes, but only on opposite-day.
The platform in question handles mobile device management, identity brokering, and mobile email management, among other things. It would certainly align with the reports about Treasury and other departments having email compromised.
https://www.securityweek.com/russian-hackers-exploiting-rece...
“The exploitation via command injection led to installation of a web shell and follow-on malicious activity where credentials in the form of SAML authentication assertions were generated and sent to Microsoft Active Directory Federation Services (ADFS), which in turn granted the actors access to protected data,”
- a good, tested key rotation practice will reduce the impact of any accidental exposures. It'll also help tremendously when staff changes, and you can do things like "rotate all keys every quarter"
- separate API keys for different services and environments means that even if your staging network-layer key is exposed, it can't affect your staging database (= data), nor your production database (= critical data).
- I've used "search for AWS secrets" as a pre-commit and also CI hook with great luck. I've even accidentally added a secret myself :) but with the checks the error was immediately found. No problems!
* Don't blindly add everything to git.
* Check staged files before commit.
* Store secrets separately, outside of your git folder (e.g. use another folder, service or environment variables).
* If it must be in your git folder, then make use of .gitignore and store keys, etc separately from code in files with a unique file name pattern.
It's possible to monitor them while pushing code, and to have a monitoring system deployed to track them.
I was thinking of open-sourcing our GitHubSniper tool, it basically has API regex from nearly a thousand SaaS and PaaS services and frequently posted sensitive strings, can be deployed via docker ;)
The only thing that stops me from releasing it is I have consistently earned more in bounties than the incentives of actually releasing it publicly. But will eventually release it :)
Nevermind...I found the docs. :-)
Otherwise, secrets as environment variables, do not write them down in files, and change them regularly.
If you change them too often then it's a maintenance problem. You might even take an outage because you have mismatched credentials (i.e. application or database had the database credentials updated but the other one didn't)
Changing them regularly IS necessary. So if you're going to do it you should use a credential service like Hashicorp Vault.
Don't use GitHub. Guaranteed to have 100% efficacy.
I add this hook which checks for passwords, high-entropy strings, etc. and has tools for managing false positives: https://github.com/Yelp/detect-secrets
If you are super paranoid you air-gap everything.
Not perfect but gets you maybe 80-90% of the way.
Here is their comparison of pre-commit hooks and other tools. It's current and comprehensive. https://github.com/duo-labs/secret-bridge/blob/master/TOOLS....
If you instinctively want to write
shiny_api_key = 'SOME.KEY.BOSS.EMAILED.ME'
Force yourself to instead take the extra few minutes to do
# my-cloud-vendor key-store create shiny_api_key SOME.KEY.BOSS.EMAILED.ME
and then write
shiny_api_key = key_store_access('shiny_api_key')
This way, when six hours later you check that code in, to GitHub, and go home for the weekend, you won't get a call from the boss ruining your Sunday lie-in, "WTF Did you upload our $5000 shiny API key to GitHub? You are fired".
This habit also allows for correct separation as an organisation grows, when you're big enough you want it to be possible for somebody to write code against test systems with test API keys, and that same code to work as written in production, even though production API keys are not accessible to the coders. Using key stores to mediate makes this much more likely to actually work.
Build in the secret retriving function from day one. Have it grab from an environment variable and it will be 100% seamless from development to production. Add more complexity/security if and when you need it.
This can later be leveraged to allow for zero-downtime credential rotations for running services, which allows for seamless use of ephemeral credentials, which is what we should all be moving towards.
The right answer here (as stated in sibling comments) is to install a mechanism into the build. A push that accidentally commits credentials should break the build. How it goes about doing that is up to you, but the specific way I've done that in the past is a git hook [1] specifically the pre-commit hook. It looks like there's a reasonably well-updated package by Yelp to help do this kind of thing [2].
[1] https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks [2] https://github.com/Yelp/detect-secrets
These types of hook are brittle. If you already knew you were supposed to do it the way I wrote, you get a hook failure, you say "D'oh" and you do it properly. But if you aren't trained to do anything in particular, this type of hook is a behavioural branch. Maybe it's 0845, your mentor just arrived, the perfect time to ask what the diagnostic means. "You need to use a dedicated key store, let me show you" she says and you just learned an important thing.
But maybe it's 1845, you're the last one in the office. You promised to raise a PR for tomorrow, but the hook fails. You try some things, and eventually discover that it hides well enough if split into four strings. You commit, click to raise the PR and go home.
And the person who reads that PR will see immediately tomorrow it's unacceptable because it has an obfuscated API key, but alas if it's a real working key that's too late.
You raise a great point which is that these kinds of mechanisms are only as strong as the cultures that support them. If the team doesn't wholeheartedly digest and commit to the new mechanism, it will at best achieve only surface level improvements.
Getting your team's culture to that point is a step zero of sorts -- completely necessary precondition and you can't do much without it. And that can end up with people paying lip service to the letter but not spirit of the mechanism.
First off, I'm pushing the principle that secrets should only be issued, created, stored and distributed by the secret management system, vault in our case. If you never see a secret, and only instruct the automation how to handle it, it is much harder to leak the secret. Ideally, in production, no human should ever need to see or access the secret.
Second, you should have good rotation procedures. Something will eventually leak, or someone will leave on bad terms and you need to be able to handle that. This can be simple - an HA infra service can just reload the core credentials from vault and be done, or somewhat harder, e.g. if customers need and use those passwords.
And finally, you should have entropy scanners like trufflehog or github scanners as linked in other tools - both in you precommit hooks (so you cannot commit the secret) as well as in your CI & repository host, so someone who disabled the security scanner "because it was annoying" still gets caught.
And I guess finally, whiskey for that friday evening someone still manages to leak something important. Some terminations at work have not been fun.
This is also what he said in the Q&A session:
> I think the attackers must have used the same approach as the FTP server was open & credentials were not strong enough. But it was a sophisticated attack as the binaries were signed.
You have one guy saying he found an FTP credential in github and on the other hand a nation-state attack and putting two and two together with no proof or reasoning other than they both happened at some point in time.
2019.
some actors who benefit from scale are purposefully trying to make Github fit that role, to gather, search and mine the work of others without paying or giving back.. but due to the distributed nature of git itself, there are alternatives
If so, maybe that's a worse situation? After all, if it's harder for attackers to scan, it must be harder for"white hats" looking for leaks to report?
If anything, this might make companies increase the risk they associate with open sourcing their code. This wouldn't have happened if solar winds were not uploading anything at all to github...
Code needs to be scanned for secrets even in closed repositories because of insider threats. But opening them up guarantees your mistakes will be caught by, ah, volunteer sysadmins.
I understand you're pointing out that fuzzing the distinction is a way for someone with an anti-open-source agenda to advance their efforts, just trying to make the point explicit.