Hacker News Reader

Comment by Analemma_ | Hacker News Reader

This is becoming such a common story that I'm starting to think it's a little irresponsible of both Amazon and GitHub to not react better to it. Clearly Amazon is capable of detecting when this happens as evidenced by their rapid response; when they see AWS keys published to GitHub, they should block the account, or at least prevent it from spinning up any new instances. They can then contact the owner over e-mail to follow up.

Same thing with GitHub. When they see an AWS key, they need to block that push and send back a message saying "If you're really, really, REALLY sure you want to do this, add .allowawskey at the root of your commit and push again". There's no excuse for letting people shoot themselves in the foot like this over and over.

(Edited for typo)

View on HN

Is it really on the web service companies to make sure you (the developer) don't do anything stupid?

The article author could have avoided a lot of headache if he simply checked the repo before walking away. Red flags should have gone up as soon as he received the first weird Amazon messages, but he still didn't think to check his own work.

The author calls this a "Data Breach" but in reality it was a "Data Leak".

Amazon would save some money by doing this, since they usually reimburse "victims" of public AWS keys (at least on the first instance) and eat the cost.

They don't "reimburse" anything. They only cancel the due amount. The only thing they lose is their wasted computing power (and some support costs) and I think the RAW computing power is very cheap to them, especially considering their scale. The cost in terms of support for the victims is probably bigger but still not significant.

It's still a cost that could be mitigated with a couple hours of someone's time.

Maybe not, but it think it is the web service company's responsibility to alert and contact you on suspicious activity ... like a sudden $6,000 jump in service usage of spot instances in regions you have never used before.

I mean a quick automated email would be enough for you to take action. The way it is set up now, you pay for it and humbly beg Amazon to deduct those charges if they see fit.

I agree, and it seems this sort of thing is already in place on AWS, but the author didn't heed the first two warnings. Short of just blocking things from happening and risking inconveniencing other customers, I don't see much more AWS could have done that would be reasonable.

You can easily set up billing alerts in cloudwatch to do this. I have done so. I agree by default it should prompt you to do so during the aws on boarding process.

Is it really on the web service companies to make sure you (the developer) don't do anything stupid?

Yes. The web service company should look at it as basic security - if people are using their service to discover private information and use it to defraud their users, even if the users have been stupid enough to upload that information publicly, the company should do what they can to mitigate the problem.

If I post a photo of my credit card on Twitter I would expect my bank to limit the damage as much as possible by declining obviously fraudulent transactions. Ultimately it's my own fault, and I would be liable for the costs, but that doesn't mean the bank should just let it happen if they can do something to stop it.

> mitigate the problem.

This seems like very minor engineering effort on behalf of GH - simple regexes, even.

To detect the key, sure. But then there's the interface design to inform the user, hooking in to an email to notify the user if they're committing via git, a system to disable the option if necessary, monitoring the AWS format in case it changes, dealing with other formats that are requested by AWS's competitors, and so on.

I don't think this would be a minor undertaking or simple in any way, but it is important enough that a provider (and target) the size of GitHub should still do it.

> Is it really on th web service companies [...]

Yes. Or, it's not so much a responsibility as it is an opportunity. I'd choose the provider that stops me from shooting my foot.

I assume AWS has spending caps for each account. My credit card company blocks my card and calls me if I spend in a suspicious pattern. And so on.

GitHub is a service. Its not their job to offer me issue tracking, or pull requests, or really much of anything. They CHOOSE to offer me things, and if those things are interesting, I CHOOSE to pay them. As users, we can certainly declare that we believe AWS key detection to be of foundational importance to a code sharing community, and signal them to thus take action or risk losing us as paying customers. I would certainly feel better if this was a feature, given its apparent ease of implementation and huge head aches it would solve.

GitHub is a value added service.

Making sure I don't do something stupid and expensive is a value add.

It's not "on" them but it would behoove them to perform better than this tale suggests. And they are not doubt capable of such.

Or just waive the charges.

There's a huge amount of value in a service that insulates you from mistakes like this. You really, really have to know what you're doing with EC2 because it is a loaded gun.

Personally, I don't like using things that are covered in spring loaded booby traps that require eternal hypervigilance in order to avoid nasty financial consequences. Certainly not when deploying something that would be right at home on something like Heroku.

> You really, really have to know what you're doing with EC2 because it is a loaded gun.

That's a fair point, but keep in mind the article author was given several warnings that something was wrong, but failed to correct the problem each time (which resulted in the huge bill he ultimately received).

At what point is the responsibility on the author here?

The author deleted the old key and created a new one and changed his password as soon as he was notified. How did he fail to correct the problem? What else should he have done?

> The article author could have avoided a lot of headache if he simply did not check in secrets into any source control system.

FTFY

It baffles me that people actually do this. Passwords and secrets DO NOT BELONG THERE. Even your 'if he simply checked the repo' statement is in my view completely crazy. No you should not have to worry about that, because that should not be there at all. Leaking your source-code should not result in a system-breach at any point.

> or at least prevent it from spinning up any new instances

This is the answer I would like to see implemented. If they can detect the key they can put a freeze on the account so that only existing resources are available.

They should also have a timeline of all created instances in every zone so you can see what has happened in the last x hours and kill all instances at once.

Two things would make controlling these situations easier:

* Allow me to see all instances in all regions from a single screen

* When accounts are first created for AWS, set up some sane default IAM policies and/or hire some UI designers to fix that godawful mess.

This is a really good point. There was another post recently linking to a search for id_rsa files (SSH private keys) on GitHub. Instead of allowing people to publish sensitive things like that accidentally, GitHub really should require an extra confirmation.

I wonder if you actually read the story. The author displays incompetence in technology administration at multiple steps.

From committing the basic design error of using hard-coded credentials at all; to committing them to any kind of repository; to failing to check on the visibility of that repository; to giving those credentials privileges they did not need; to failing to act effectively on warning messages.

And all this, apparently, is laid at the feet of Visual Studio.

I have no love of the Microsoft ecosystem, but the author does not acquit themselves well in this incident. The arrogant claims of seniority in the opening paragraph seem ironic in this light.

> all this, apparently, is laid at the feet of Visual Studio

I wonder if you actually read the story.

He admits several faults including most if not all of those you mention.

So what? Admitting fault isn't absolution. He really does go out of his way to blame the tools.

The tool has a huge bug! Even if he wasn't putting credentials in there, it could still have led to sensitive source code getting out or something!

Article updated - VS wasn't the issue. He had installed a 3rd party github extension (from github themselves) that overrides the built-in Git support in visual studio... bug was found, confirmed and fixed yesterday though, so a quick turnaround.

Exposing an AWS key through a public repo happened to me a few years ago and AWS support called me to confirm whether I was aware of the suspicious activity, when I stated I do not know what's going on they immediately blocked the account and told me I wouldn't be responsible for the cost. Mine was actually much worse, the account was billing a thousand dollars every hour. They told me I'd have to sign up again with a different account.. I'm surprised this process was not followed in author's case.

obviously someone is good at detecting it considering 6.5k in usage got racked up with data scraped within 10 minutes of commit

The quick bitcoin mining spin-ups, and the initial email from Amazon about the key going public, are (probably) thanks to the github events firehose.

https://github.com/hootware/github-firehose-node

http://blog.scalyr.com/2013/10/exploring-the-github-firehose...

It sounds like it happened the next day, after he changed the keys.

AWS do do things. They have service limits and routinely waive charges from abuse events like this.

I do wonder if GitHub's events firehose has benefits that outweigh all the compromised credentials, though. Not just limited to AWS keys.

> They have service limits and routinely waive charges from abuse events like this.

That's almost worse though. Maybe I'm being unfair here, but the fact that they will waive charges immediately if you ask, but don't just block them pre-emptively, tells me that they're hoping at least some customers (with deep pockets and automated billing that no one pays attention to) will just pay the bill and never notice. Maybe they're making more money from that than they're losing by waiving charges on compromised accounts for customers who ask.

Devils advocate, wouldn't you rather have illegitimate charges waived after the fact that legitimate charges declined because their algorithm threw up a red flag? Something like being posted to HN could cause a huge spike in charges for your account in a similar way as leaking your keys and it would be beyond frustrating to see that opportunity disappear because Amazon saw your bill spike.

Don't be ridiculous. AWS want legitimate revenue, not brand damage.

You really want GitHub analyzing your code? REALLY? If the internet found out they were analyzing every push the backlash would be 10x what it is now.

Even if it's public... That's still a Gmail-esque situation that would immediately raise privacy concerns.

Developers should pay attention to what they're doing. Your AWS keys shouldn't even been in your private repo. You should find another way or you shouldn't be using a third-party for source control.

Looks like someone is analyzing it anyway. May as well be to your benefit.

It would probably be easy for Amazon to offer an automatic key blacklisting feature....

Natural selection at work, do not disturb.

"letting people shoot themselves in the foot like this over and over"

Stop doing it in the first place, problem solved. I don't see why GitHub or Amazon need to hold my hand through a serious process. It would diminish the importance of what you're doing.

Also, this guy didn't just use AWS keys...he clearly used AWS master keys...which has not been the recommended course of action for AT LEAST THREE YEARS...if he had used a proper IAM user with only the access he needed, this wouldn't have happened. Sorry, but that was just irresponsible.

"I don't see why GitHub or Amazon need to hold my hand through a serious process. It would diminish the importance of what you're doing."

No it wouldn't, not at all. Not unless you're one of those, "REAL MEN us {$editor}" people, in which case no one cares about your opinion.

Reply on news.ycombinator.com