Hacker News Reader

Comment by eksith | Hacker News Reader

This sort of thing has happened before to another company, but that was a case of human error compounded by the fact that there were no software safeguards in place.

http://dreamhost.com/dreamscape/2008/01/15/um-whoops/

At the very least, it would be wise for any company to implement billing features that will under bill rather than overbill. Under billing at least gives you the option of notifying users later (or you just eat it). Overbilling can sour a professional relationship to the point of ending it.

View on HN

Unfortunately when automation goes wrong, it usually tends to do too much - double billing, aggressively canceling things, etc.

Having built several automation systems that bill millions of dollars, I can say that it's quite challenging to build things that have no visible output, run by themselves and often sit there for years without being touched, while everything changes around them!

It's not sexy work, but when it goes wrong the shit really can hit the fan!

Here's a choice piece of code we have in production along the lines of what you suggest:

    if prorated_amount > (cost_per_month * 2)
      raise "critical! we were about to prorate-charge someone #{prorated_amount}"
    end

It's never been triggered, but this is probably the only place in our system where we're actually responsible for an exact amount that ends up on an invoice, so having a reasonable upper limit seems like a no-brainer.

shouldn't it be >=, to catch double-charging?

What if this code is called multiple times in quick succession? Based on the fact that I got a dozen separate emails over the span of 10 minutes, I'm guessing Twilio's overcharging happened in a series of transactions, not all at once.

This code -- while good for catching bad single calculations -- wouldn't catch that.

Well I don't think he was saying "cut and paste this exact code into your codebase" but rather that it is possible to write safeguards such as this... in this case, perhaps checking whether an account has already been billed this month, or whether there is already a transaction pending, or...

We can't prevent every mistake but we can make sure the same one doesn't happen again.

Reply on news.ycombinator.com