Hacker News Reader

Comment by maxgashkov | Hacker News Reader

That’s extremely hard to design, at least with the current state of what AWS bills and does not bill.

Example: let’s assume you’ve set the cut-off budget too strict, spun off another shard for your stateful service (DB for example), it received and stored some data during the short window before some other service brought whole account over budget (i.e. paid egress crossed the threshold).

To bring VM and EBS charges to 0 (to implement your idea of ‘shut down everything’) AWS will have to delete production data.

While it may be OK when you’re experimenting, it’s really hard to tell in automated way.

So, to fully comply w/o risking losing customer data, AWS will have to stop charging for not running instances and inactive EBS volumes which most definitely bring on many kinds of abuse.

—

There may be some other way to do this, maybe mark some services expendable or not, so they are stopped in the event of overspend.

View on HN

A complicated solution is not what people are really asking for though.

What I and I expect most people want is a cap which then spins down services when they reach that cap. Nobody is going to care if the cap is set to $1000 and the final bill is $1,020. The problem being solved for is not wanting to have to ever worry about missing an alert and waking up to a bill that is a factor or two beyond expectations. I can afford my bill being 10% or even 40% above my expectation. I can't afford my bill being 500% off.

I do understand that, but there are services that are still billed for when ‘spun down’. To stop getting the bills they have to be terminated.

The solution seems to be to implement ‘emergency stop’ when whole account is put to pause but no state is purged. And then you’ll probably have a week or two to react and decide if you want larger bill, salvage the data or just terminate it all.

> So, to fully comply w/o risking losing customer data, AWS will have to stop charging for not running instances and inactive EBS volumes which most definitely bring on many kinds of abuse.

Another option would be to "reserve" them in the budget. That is, when for instance you create a 10 GB EBS volume, count it in the budget as if it will be kept for the whole month (and all months after that). Yes, that would mean an EBS volume costing $X per day would count in the budget as 31*$X, but it would prevent both going over the limit and having to lose data.

This creates a surprising situation.

We have a batch process that uses S3 as a way to temporarily store some big datasets while another streams and processes then and, once complete, removes them. This takes like an hour.

So our S3 bill could be, let’s say, $10/mo. If you went and looked at your S3 bill and saw that and thought setting a $20 cap would give you reasonable headroom you’d be sorely surprised when S3 stopped accepting writes or other services started getting shut down or failing next time the batch job triggered.

Under this system, the budget and actual bill need to be off by a factor of more than 10 ($240). And this also doesn’t stop your bill being off by a factor of 10 from what you “wanted” to set your limit to. More than the $200 under discussion.

I think there's a good argument that they could do better. But there's also probably an argument that harder circuit breakers would result in posts like "AWS destroyed my business right when I was featured in the New York Times"--including things like deleting production data. I'm sort of sympathetic to the idea that AWS should have an experimental "rip it all down" option and kill all stateless services option but that adds another layer of complexity and introduces more edge cases.

Merely being on free tier is a signal that you do not want a $27,000 bill. There is no excuse for what AWS does here, and it is clear they use this clusterfuck as a revenue stream.

They can simply add a nuclear option. Ordinary business probably won't enable that option but individuals can and should set it up.

It’s a lot easier for them to refund the individuals who make mistakes than deal with the fallout and bad press from the small businesses they’ve destroyed when they incorrectly enable that option.

Yes, a useful last resort safeguard would need to be more granular than just "turn everything off", at least if we're talking about protecting production systems rather than just people learning and wanting to avoid inadvertently leaving the free tier or something like that.

Still, it's not hard to imagine some simple refinements that might be practical. For example, an option to preserve data in DB or other storage services but stop all other activity might be a useful safety net for a lot of AWS customers. It wouldn't cut costs to 0, but that's not necessarily the optimal outcome for customers running in production anyway. It would probably mitigate the greatest risks of runaway costs, though.

Reply on news.ycombinator.com