How a bug in VS2015 exposed my source code on GitHub and cost me $6,500
humankode.com
humankode.com
Same thing with GitHub. When they see an AWS key, they need to block that push and send back a message saying "If you're really, really, REALLY sure you want to do this, add .allowawskey at the root of your commit and push again". There's no excuse for letting people shoot themselves in the foot like this over and over.
(Edited for typo)
I do wonder if GitHub's events firehose has benefits that outweigh all the compromised credentials, though. Not just limited to AWS keys.
That's almost worse though. Maybe I'm being unfair here, but the fact that they will waive charges immediately if you ask, but don't just block them pre-emptively, tells me that they're hoping at least some customers (with deep pockets and automated billing that no one pays attention to) will just pay the bill and never notice. Maybe they're making more money from that than they're losing by waiving charges on compromised accounts for customers who ask.
This is the answer I would like to see implemented. If they can detect the key they can put a freeze on the account so that only existing resources are available.
They should also have a timeline of all created instances in every zone so you can see what has happened in the last x hours and kill all instances at once.
* Allow me to see all instances in all regions from a single screen
* When accounts are first created for AWS, set up some sane default IAM policies and/or hire some UI designers to fix that godawful mess.
The article author could have avoided a lot of headache if he simply checked the repo before walking away. Red flags should have gone up as soon as he received the first weird Amazon messages, but he still didn't think to check his own work.
The author calls this a "Data Breach" but in reality it was a "Data Leak".
I mean a quick automated email would be enough for you to take action. The way it is set up now, you pay for it and humbly beg Amazon to deduct those charges if they see fit.
Yes. The web service company should look at it as basic security - if people are using their service to discover private information and use it to defraud their users, even if the users have been stupid enough to upload that information publicly, the company should do what they can to mitigate the problem.
If I post a photo of my credit card on Twitter I would expect my bank to limit the damage as much as possible by declining obviously fraudulent transactions. Ultimately it's my own fault, and I would be liable for the costs, but that doesn't mean the bank should just let it happen if they can do something to stop it.
This seems like very minor engineering effort on behalf of GH - simple regexes, even.
I don't think this would be a minor undertaking or simple in any way, but it is important enough that a provider (and target) the size of GitHub should still do it.
Yes. Or, it's not so much a responsibility as it is an opportunity. I'd choose the provider that stops me from shooting my foot.
I assume AWS has spending caps for each account. My credit card company blocks my card and calls me if I spend in a suspicious pattern. And so on.
Making sure I don't do something stupid and expensive is a value add.
Or just waive the charges.
Personally, I don't like using things that are covered in spring loaded booby traps that require eternal hypervigilance in order to avoid nasty financial consequences. Certainly not when deploying something that would be right at home on something like Heroku.
That's a fair point, but keep in mind the article author was given several warnings that something was wrong, but failed to correct the problem each time (which resulted in the huge bill he ultimately received).
At what point is the responsibility on the author here?
FTFY
It baffles me that people actually do this. Passwords and secrets DO NOT BELONG THERE. Even your 'if he simply checked the repo' statement is in my view completely crazy. No you should not have to worry about that, because that should not be there at all. Leaking your source-code should not result in a system-breach at any point.
https://github.com/hootware/github-firehose-node
http://blog.scalyr.com/2013/10/exploring-the-github-firehose...
Stop doing it in the first place, problem solved. I don't see why GitHub or Amazon need to hold my hand through a serious process. It would diminish the importance of what you're doing.
Also, this guy didn't just use AWS keys...he clearly used AWS master keys...which has not been the recommended course of action for AT LEAST THREE YEARS...if he had used a proper IAM user with only the access he needed, this wouldn't have happened. Sorry, but that was just irresponsible.
No it wouldn't, not at all. Not unless you're one of those, "REAL MEN us {$editor}" people, in which case no one cares about your opinion.
From committing the basic design error of using hard-coded credentials at all; to committing them to any kind of repository; to failing to check on the visibility of that repository; to giving those credentials privileges they did not need; to failing to act effectively on warning messages.
And all this, apparently, is laid at the feet of Visual Studio.
I have no love of the Microsoft ecosystem, but the author does not acquit themselves well in this incident. The arrogant claims of seniority in the opening paragraph seem ironic in this light.
I wonder if you actually read the story.
He admits several faults including most if not all of those you mention.
Even if it's public... That's still a Gmail-esque situation that would immediately raise privacy concerns.
Developers should pay attention to what they're doing. Your AWS keys shouldn't even been in your private repo. You should find another way or you shouldn't be using a third-party for source control.
We're only one sentence in and I'm already very, very nervous about the author's hubris.
> What could be done to prevent this?
> Always test new version control GUIs before using them in the wild. There could be a bug that could expose your data.
> Encrypt access keys in config files instead of just leaving them exposed in the config file.
> Better yet, move access keys to a seperate config file, and exclude this from Git deploys with a .gitignore.
The fact this supposedly security experienced and conscious developer thought it was okay to have any kind of access credentials stored in a git repository is glaring.
You should consider all data in any repository public for security reasons. Publishing to github publicly made it quite obvious - but he could have been just as screwed over by people he gave access to the private repositories and people who have broken into github and can directly access the repositories.
Sure the numbers on both those groups are lower than the numbers of people apparently scanning public github for access keys - but the severity of the risk of keeping access keys and other secrets in your revision control are the same.
How do decrypt them when you app re-launches? How do you decrypt them automatically on every instance?
http://docs.aws.amazon.com/AWSSdkDocsNET/latest/V3/Developer...
Also putting keys in (even a private) cloud-hosted repos is at least a step under what I'd describe as "very security conscious" for someone who self describes as working in the financial industry.
2. He pushed code to a new repository without verifying the security configuration of the created repository (granted the tool made it easy to do this but you should create a repo, verify it, then push code).
3. He used his master AWS account key/secret in the code which gave global access to everything.
4. He didn't use IAM credentials with a restrictive policy set to just access the resources required.
No, clearly not security concious.
I'm a solution architect in the financial services industry and have been for 16 years. Never do I assume I know what the hell I'm doing.
Here's how the sync dialog appears in Visual Studio 2015 without the GitHub extension: http://i.imgur.com/EGjYziR.png
After installing the GitHub extension: http://i.imgur.com/shRBwXM.png
Try it yourself, then complete these fun, but completely optional activities:
1. Re-read all comments on this thread using your best Andy Rooney voice.
2. Wish that the author had comments on his blog.
Ok...
> Better yet, move access keys to a seperate config file, and exclude this from Git deploys with a .gitignore.
No shit?
He didn't get a $6.5k bill because of a bug in VisualStudio, he got one because he didn't finish setting up his AWS account. Regardless of the public/private mixup, checking in AWS credentials to any repository is just bad practice and creates a huge risk of the exact situation that happened in this story.
> He didn't get a $6.5k bill because of a bug in VisualStudio, he got one because he didn't finish setting up his AWS account. Regardless of the public/private mixup, checking in AWS credentials to any repository is just bad practice and creates a huge risk of the exact situation that happened in this story.
I agree
P.S.: never ever put keys into repositories. Except those repos are deployment config only.
It's similar to driving your car too close to the car in front - you will get away with it most of the time and get lulled into a false sense of security, but eventually it will catch up with you.
Don't be one mistake away from a disaster. Make room for at least two.
Either the author is just mistaken/lying about that point, or there's another leak he doesn't know about, or there's a serious privilege escalation vulnerability in AWS.
If one wants to use version control to track credentials, then this makes sense. Why would one want to do that? In order to track versions of one's production system. This enables things like one-shot rebuilds.
Now, it's obvious that such a production repository should be separate from one's development repo(s). And one could make a decent argument that it should never be pushed someplace like GitHub anyway, even as a private repository.
Wild speculation, but this is a side effect of "developer" centric operations. Everything is done via code, and developers are instinctively trained to check in code.
And I don't blame him for making that mistake. That kind of bug should have been caught in QA.
On the other hand, I'm paranoid enough about things that I wouldn't push unsecured access keys to any server I didn't control. Even if I was paying for a private repo.
edit: my guess, some kind of crypto currency mining?
2) Someone other than the OP should pickup/waive the tab.
"I posted my private AWS keys by mistake on github, a spider got them, I got a huge bill"
Yeah, you don't say
Is it, though? Committing them to the same repository as one's code lives in, sure, but committing them to a separate production-deploy repo seems okay to me (although I'd much prefer that private repo never to hit a centralised service like GitHub).
Yes, it is. Period. Full Stop. Don't ever check access keys into any repo, public, private or even self-hosted. AWS needs to do a better job of making you realize that access keys are like 100-year-old sticks of dynamite and should be handled with an equal amount of care considering they can cause a similar amount of damage.
To their credit, they basically noted this when they changed the way access keys are handled in the command line tool[1]. Quoting:
An important point is that the default location for the credentials file is a user
directory. It's no longer part of a project file structure, such as an app.config file
(.NET) or .properties file (Java). This can enhance security by allowing you to keep the
credentials in a location that's accessible only to you, and it makes it less likely that
you'll inadvertently upload credentials if you upload a project to a developer sharing
site like GitHub.
So, how do you propose to handle version control of your exact production configuration? Do you think that recording one's production configuration is a valuable (or even useful) technique? Do you propose to have a single file with production configuration relying only on point-in-time backups for history? That gets us back to the bad old days…
A local or self-hosted repo is 'a local location that's accessible only to youation that's accessible only to you.'
That looks pretty cool, at least at first. I'll have to take a deeper dive into it.
I like that it looks to be all shell scripts. Wish it were sh rather than bash, but c'est la vie.
Version control is perfectly acceptable for that, but the key point is that access keys aren’t part of your production configuration. I personally don’t allow my team to have long-lived AWS access keys that have access to production. Production is only changed via an assumed role requiring MFA or an IAM instance role.
If this kind of setup is too complex to create on your own, solutions like Vault (https://vaultproject.io) can make it much easier to get right.
Why not? If your production configuration is stored as static-file in version control, then either those credentials have to live in that version control, or they have to live outside it, as a separately-backed up file of configuration variables, which are then fed to a static-file-generating tool, which means a) that one has to select a templating system and maintain those separate templates and b) that there's another step involved in developing the prototypes for those templates. This isn't a bad course of action, of course, but it does have its costs (just like storing production configuration—including credentials—in version control has its costs as well as its benefits).
> I personally don’t allow my team to have long-lived AWS access keys that have access to production.
Of course! Developers shouldn't have access to production systems, period. But operators can, and must, and I've no problem with them having access to their own version-controlled repository of information that no other team can access.
> Production is only changed via an assumed role requiring MFA or an IAM instance role.
Of course! I'm certainly not arguing for personal credentials to be stored in an operations repo.
Generalizing here (and for the record I agree with your point), is that to a developer, source code is as valuable as anything else, and if a private repo is secure enough for source code, it should be secure enough for access credentials.
I speculate that this is a by product to moving towards a developer-centric engineering culture.
> I am certainly not innocent here and some mistakes were made on my part. [...] To this end, having encrypted access keys or excluding configuration settings from GitHub would have prevented the AWS charges - and this is certainly the approach I would take from now on.
Also if you use a proper IAM profile locked down to specific resources then you wouldn't expose your entire account. The author stated he didn't use EC2 so why wasn't the key/secret pair an IAM account with a policy set for minimal access?
What I'd worry more about is that the VS bug exposed private source code, data and proprietary intellectual property.
We use github and I worry every day someone will public fork one of our repos by accident. That would be a grave fuck up but it's waiting to happen. We should have stuck with centrally controlled active-directory integrated SVN from a security perspective (even if it is a pain in the ass).