Hacker News Reader

Comment by falcor84 | Hacker News Reader

Agreed. They say in the postmortem they it was protection against crypto mining, but what kind of weird reason is that?! If I want to pay for 10 instances mining crypto, why the hell wouldn't I be allowed to do that? I don't see why they should block any workload as long as the credit card details are valid.

View on HN

It's a customer protection method. Most cryptominers are not using accounts they pay for. They compromise customer accounts and spin up resources. If you aren't proactive about communicating this to customers or blocking it, it can be quite some time before the customer notices and almost all customers will request a refund - even when the attack is a compromised password / successful phish on the customer's side.

Additionally, all cloud providers operate on various models of over-subscription. It is not in anyone's (customer / provider) interest to allow the full consumption of resources when the activity is fraudulent.

As you can see in the post-mortem, they are fine with the usage. They have a process and flag to allow legitimate customers to use their resources. However, based on previous experience at another cloud provider, I would bet that over 90% of those automated hits are correct.

This was bad support. They know that and they seem to be making the right moves to fix it. Fraud is bad for everyone and has to be combated. Not doing so can raise prices and kill a business like DO. I'm sure they feel awful that a customer was so poorly impacted, but the error wasn't in the first ban, it was everything after that.

If it was just the first point, the customer should be able to confirm that the activity was intended without even going through human review. It should be like when your bank texts you to confirm an odd transaction. They don't simply lock your account.

It sounds much more like it was the second point, which is unsettling. It's one thing to plan your pricing based on the assumption that most customers won't maximally-utilize. It's another thing to enforce a soft-limit that's vague and below what was advertised. I'd much rather have a lowered, known limit than whatever this is.

I totally agree, but unfortunately my bank (a major U.S. bank) does block a transaction and sometimes lock my credit card completely when they think the transaction is suspicious. There’s no confirmation mechanism, I have to call them to get the card unlocked. Of course, this usually gets resolved within five minutes (except that one time when I had to renew a .ng domain, and the Nigeria-originated transaction got auto-blocked three times in a row, and eventually the case had to be escalated to override their security mechanism entirely), not 29 hours.

> They don't simply lock your account.

Capital One did this to me once, and refused to restore the use of the blocked account even after I immediately called them and confirmed that the transaction that triggered the block was not fraudulent.

Part of the whole issue here revolves around shared hosting in my opinion. Host hardware is so oversold that one customer utilizing 100% CPU is so impactful to a handful of other customers that it's not allowed at all. I have seen providers that has terminated services for less than 100% CPU usage, a constant 90% is enough on some of them. But due to the profit margins and shared hosting, providers are able to charge incredibly low prices per instance and be able to oversell their hardware sometimes as much as 10 to 20 times. That's as many as four hundred customers on a box that should maybe have 20 if it weren't oversold it all. In this case it really is an instance of you get what you pay for. The service we provide is no oversold hardware and all dedicated plans. Some people are initially very turned off by the pricing but the ability to allow customers to mine if they wanted to and not affect a single other customer on the platform giving each customer the same experience regardless of any other one images resource utilization, leaves too much happier customers even if smaller profit margins for us. At the end of the day customer experience and support provided are two of the most important factors in running a hosting provider. While I disagree with aspects of digital oceans business model as a shared hosting provider, I do think that the response to this was more than appropriate and better than would be expected of a lot of shared hosting providers, provided they actually implement any of the things talked about in the response.

When you say we/us as a more expensive, but dedicated alternative, what is the cost difference as a percentage for say a small project?

Edit: found your site, looks like you’re cheaper than aws at a glance

It was in combination with a lack of payment history. So if they had been paying it would not have triggered but they had been working off of credits instead. I think this point addresses your concern that paying customers should be allowed to mine.

I ran some really long compute jobs on GCP (100% CPU for weeks across many vCPUs) with credits without getting flagged. I was evaluating FFTW performance for a project. Perhaps GCP could tell I was calling into FFTW and not mining so they decided it wasn't fraud?

Perhaps, or they viewed credits as payment history? I’m not defending the algorithm as even DO has said it was a false-positive. I just wanted to point out that this wasn’t an attack by DO on paying for crypto. That it specifically was trying to look at non-payers.

It makes sense for a company like DO to not allow crypto miners to use credits. Or else they would develop elaborate systems to create fake accounts and spin them up to mine.

Google can afford to eat the cost and perhaps has better heuristics to detect mining. And they definitely have better data to detect a single user signing up for multiple accounts.

Reply on news.ycombinator.com