"I don't know how my key leaked. It's a 9yr old account, under utilized + running old projects.
AWS do have alerting systems you can use to catch overspend. You should.
They didn't exist when I signed up. And with 200 links in their main menu, they're not easy to find."
Don't blame the user for what is essentially Amazon's problem. They don't have a way to set a hard limit on spend. They don't even do basic velocity tracking apparently.
Every time I see a story like this (and there have been many), Amazon always ends up dropping the bill. Idk what this one hack on Lambda actually cost Amazon, but I’m guessing it’s nowhere near $45k.
Yeah, one key issue is that it's deceptively easy to set something up in AWS, but there's actually a ton of customizability, complexity and potential gotchas lurking in the background.
They've essentially SaaS-ified sysadmin, network admin, DBA, etc. roles. So relatively casual users are pointing, clicking, copying and pasting like they're just navigating any ol' app, but beneath the surface they're commanding potentially massive/unlimited resources with the attendant costs those resources impute.
Makes you wonder whether it's just too much power to casually have on offer.
I recall a VPS that i had years ago being hijacked due to Docker socket being exposed publicly (when i set one up for testing and was just learning about Docker), which was mining crypto throughout the night before i noticed it.
Imagine what would happen if there had been some sort of auto scaling in place and i had used a managed Kubernetes solution (many of which didn't yet exist back then)...
Is it Amazon's problem? Not really, apart from the bad PR that charging private individuals exorbitant sums (for them) might cause, so sometimes these bills are waived after people mess up. For the corporation at the size of Amazon, that's probably not too much in the grand scheme of things, but it's also understandable why people might be appropriately upset about not being able to set hard spending limits, merely alerts (which aren't enabled by default).
That's just the reality that we live in for now, however one can also think about why a corporation that cares a lot about uptime might not necessarily want to implement functionality that'd let resources be easily cut off when spend limits are hit or something similar, since most businesses that utilize their services might just swallow the occasional expensive bill like that anyways.
Not saying the OP has no responsibility here. Just that I don't think it is that black and white. There is plenty AWS could do to avoid this sort of thing happening. Totally fair to call them out for that. Especially when the cost of phoning support is $2-3k specifically because of the new costs on your account!
Absolute spending caps, or spending growth rate limits, would both be useful even outside of situations where a user's credential is compromised.
One example: When teaching colleagues how to use AWS tooling, I gave them full access to a sandbox account where they can learn without disrupting anybody else's infra. It would be nice to be able to place a spending cap on that account.
Another example: With things like autoscaling, it's very easy to accidentally misconfigure it to where it will happily start launching tons and tons of VMs. You might do this and not notice. If I had the ability to voluntarily opt into limits on how fast I can increase the aggregate $/second spend rate, then AWS could alert me in general when I'm doing something that exceeds that soft limit.
The OP was a victim of the hacker and not AWS.
My point is that AWS failing to implement spending controls makes this type of attack significantly more damaging, and the slow turnaround on CloudWatch spending alarms means five or six-figure sums of money can be spent before customers are alerted.
AWS can and should do more to give its customers additional control over spending as a defense-in-depth measure. This would be valuable both to limit unauthorized spending (this scenario, or a "rogue employee" scenario) and also unanticipated spending ("autoscaling gone wild" type scenarios).
AWS is a big target, and the fact that credentials can be used to spend an unlimited quantity of money means they are a huge target for criminals. Defense in depth would be wise in this situation!
Security needs to have depth. The one thing we have learned in these few decades of learning how to do online security is that any single measure can fail, and given enough scale, will fail. For anything to work, you need to be able to layer your security measures so that one failure, no matter how grave, will be fatal. AWS could add a very simple security measure here, making their service dramatically safer for their users, but they choose not to.
But the 3.4 trillion dollars have to be paid back by Monday! And take better care of your information next time. It is the duty of every customer to check cash dispenser for skimmers before using them! This is clearly written out in out general terms and conditions!