“DigitalOcean Killed Our Company” (2019)
twitter.com
twitter.com
https://web.archive.org/web/20220417154429/https://www.digit...
Anyway, I digress...
so high CPU utilization is allowed under certain cases
(Not a judgment on what happened here)
I'm the guy who made the Twitter post and wanted to clarify a few things because this Tweet keep coming back every year with the same comments.
1. About the thread:
First, I would like to say that this thread was purposely dramatic with the only goal to get a quick answer from Digital Ocean. We did not have a communication channel with them after their last ban email, and it was a way for us to promptly establish a communication through social pressure. And with hindsight, probably a bit of immature social-media revenge of creating a shitstorm on social media.
DO did not kill our company, they put us in an uncomfortable position. We did have two Fortune 500 clients at that time, but we had no critical data, just a few configuration variables that we were aware of and could probably recreate from our memory.
2. About the backups:
We had two database backup, DO automatic disk backup and a backup script that uploaded to DO object storage.
YES, IT WAS A MISTAKE to have all our data with a single provider. I've learned from this, and it was a valuable lesson for all the people that followed this story.
At the time, I was the only person with tech skills in a two-person company. My tasks ranged from UI/UX to NLP to DevOps, I had professional experience with frontend / backend, but it was my first role where I had to handle system administration. I think I did a fantastic job with it for a junior, outside not setting up multi-provider backups.
3. About what we did to get ban:
It really was a python script that had to process ~500k records and fuzzy match them against another DB with 1M records. It was pure business logic and nothing illegal / shady / or against DO TOS.
As stated in the postmortem, we did not have a payment history with DO, as we were still on free credit at this time. We had no malicious intent to avoid payment and had the mean to pay for what we used once the credit ran off.
4. About Digital Ocean:
I do not hold any grief against DO, it was a human a process error, it happens to everyone, big and small companies. What matters is that they fixed quickly, did a postmortem on it, and acted to prevent this situation in the future. Nobody got killed, no data got lost, and we all learned from this event.
Hope this clarifies the story a bit. And please stop harassing me about multi-cloud backups :s
As a person that has been watching this kind of horror stories from outside, I just wonder why people choose to make business with companies that have no human in the loop when stuff gets serious.
AFAIK, for AWS for example, when things get serious you are prompted to upgrade to a the big boi support plan (1.5k$/month) and are assigned a TAM (Technical Account Manager). In a situation like that, if you were running on AWS, a TAM or any human in general would have probably fixed the whole thing in like 30 minutes (if anything, because they would know you and that your usage is legitimate).
DO had humans in the loop, they just followed a wrong process and choosed to cut communication.
Also, the probability of loosing your account vs having to spend $20k a year just for support is a quick decision to make when you start from nothing.
Today, I work for a company with billions in revenue, and it would be unthinkable to not get an AWS Account manager.
In the end it's always about risk vs reward.
While it was only a few hundred dollars, actually using the compute I paid for in advance still resulted in a permanent ban, and Digital Ocean kept the remaining balance. It was only around ten droplets, created manually. What exactly should one do if one actually needs the compute?
Even if you do care for whatever reason, isn’t this something that’s better addressed via pricing adjustment? You won’t have any miners if they spend more money renting the instance than they get by mining.
Cloud providers would rather just not provide that service, detect and stop it before it gets too out of hand.
The handful of times I’ve personally encountered it, it was always some sort of fraud. Notices from the cloud provider were how a given company found out about it… before the surprise alerts or bills.
Pricing adjustment doesn't help if cryptocurrency mining is being used to generate income from VMs leased with stolen credit cards. Might be more useful to have a CPU limiter that eases up as the VM (or payment) ages; although if that's not public knowledge, it will lead to support requests.
Ideally, the amount charged would be a sensible reflection of real-world costs, including power usage (perhaps with occasional concessions to provide some measure of pricing stability to customers). Otherwise, it’s no surprise that customers will take advantage of that arbitrage.
> Pricing adjustment doesn't help if cryptocurrency mining is being used to generate income from VMs leased with stolen credit cards. Might be more useful to have a CPU limiter that eases up as the VM (or payment) ages; although if that's not public knowledge, it will lead to support requests.
Then the problem is the use of stolen credit cards, not crypto mining. I could just as easily spin up a botnet, or scammy gambling/warez site, or a spammy email campaign, or a myriad of other things that could negatively impact the business’s finances or reputation. The right way to address this issue is to improve the KYC process so that such cards cannot be so easily used in the first place.
Improved KYC process is easier to say than to do without chasing off customers. I'm not going to participate in much in the way of KYC in order to get a $7/month VM. If they want to see my government issued identification, I'll find someone else to sell me a VM, it's a crowded field (and I'll be leaving anyway, since they are dropping support for my preferred OS; once I decide on which of the many others to move to)
Spammy email is usually solved by restricting port 25 access. It's amazingly common to have that port blocked for new accounts in hosting. Botnet or scam sites don't really have a major cost for the hosting company; maybe you have to field a couple abuse reports, but nobody is going to blacklist your IPs for that, and they don't tend to use a lot of power or bandwidth, so it's not a big cost.
I can think of a few reasons. I'm not sure how many apply in this case, but I think all of them.
1) Many service providers want to provide a free tier. It's a fairly inexpensive way to try to hook someone who will eventually pay off in spades. This is common in many industries. The difference is a business may scale because of demand - crypto miners will not.
2) There is a difference between 100% 24/7 utilization and intermittently hitting 100% at high load times. This has to do with hardware lifespans, the ability of overcommit hardware (like ISPs overselling bandwidth or airlines overselling tickets)
3) A company may think crypto is morally/environmentally/legally/PR bad and forbid it for the same reasons as weed, tobacco or porn. Or for processing transactions for members of rogue nations.
4) Using stolen CC numbers to mine crypto seems like a lossy but effective way to turn those numbers into actual money.
That's just trivially off the top of my head. As to your other question - no. None of the issues I described (unless you think any non crypto 100% pegging is likely to persist forever) are better solved with pricing adjustments. I don't let people smoke in my home. There is no reasonable sum someone could pay me to just smoke one cigarette.
Being in the industry for years I've heard similar stories for XXXX (insert your favourite, extremely "friendly" and reliable provider).
Never build your service on single provider. At least 2, preferably 3+ for redundancy.
Second, if you have Fortune 500 companies using your service, why would you not keep offsite backups… I hope they learned from this…