bcooks

Hi cyberferret. Two quick trys to provide some clarity for you here:

1. On your vendors don't say why they took action point. I think a lot of vendors, us included, worry about the bad actors knowing the _how_ of our algorithms because they will then have intelligence on what to try and work around to avoid detection. It's a constant adversarial chess match we adjust, they adjust.

2. As for the human element being needed for the after effects I totally agree. That said, I looked at our numbers and while many of the nefarious actors simply move to the next thing, a very large percentage of the, "WTF" replies are actually from bad actors hoping that is enough to be re-enabled and to keep going for a while. Our goal is to bring more people into the after effects and give them better information to make decisions and work with customers to avoid this kind of mess in the future.

DO has a startup program called Hatch: https://www.digitalocean.com/hatch/

That is the starting place for many folks.

Hi weaksauce... sorry I must have missed your earlier question.

The account had been live for some time and in that sense had history but because of credits it didn't have payment history. As some others have commented lots of startups use credits to get their business going and depending on your usage they can last you for quite a while. Payment history indicates a willingness and capability to make payments.

Part of the issue here was what triggers the algorithm used when looking at remaining credits, payment history (none), workload deltas (the new spin ups), and effective run rate (think of that as the amount of money they would be charged for the workload they were spinning up). The bug in this case was both simple and super impactful. Raisup did nothing wrong, everything right in fact. We just blew it.

Thanks for the comment on request for download of backups or snapshot. That is a great idea, I guess we just never expected to actually go shoot a real customer and the fraudsters don't ask for their data.

This is correct. This was the primary thing we were attempting to solve for in this case and the bug in the algorithm started the chain of events documented in the postmortem.

Sorry to hear that you had a bad experience and left with a bad impression of that team. We have a number of data sciences efforts including in the core R&D group where we are growing and working to improve models in support of a number of fleet monitoring tasks

I agree on the twitter cred point. The fact that this happened in the end, personally I think it is a good thing as it highlighted a weakness we must fix.

We trust our people high-level, low-level whatever to make important decisions everyday. thats why they are here.

The "marketing communications specialists" are getting slammed a lot here, so I will just point out that they spend most of their time rolling their eyes at my crappy grammar, spelling and ludicrous number of comma splices. I don't think our goal was to sound like anything. We just wanted to lay out our investigation and the follow on work we are undertaking.

Totally agree with your point that trust is earned and we lost many peoples in the last few days. That will take time and as you say good behavior to earn back, but that is what we are committed to doing.

Correct.

Nothing was deleted or removed. The droplets were powered off and the access to them locked. once (way too long later) the unlock happened the customer had full control and access again.

Thanks for the pointer. I'm going to blame my dad/up bringing for my over use of passive voice. He will be deeply amused by this. I will read the paper and attempt to improve.

Hey there. Thanks for this feedback. I think it is important to be open honest but not blame-oriented in our review of the situation. People make mistakes and that is okay, so long as they aren't willful or due to incompetence. Neither of which was the case here. The key thing is not to create a situation where a mistake is an individuals fault. My general view is if people are making mistakes then we have done something wrong as a company and need to understand and fix the tools/training/process that led to the mistake.

The first few items on your list are actually a part of what we meant by "having billing history with us". There are a number of things we look at in that bucket. We use these items as a part of validating users before taking any action (yes, we clearly failed on this account due to the credits which is a clear bug). As far as offering things like a copy of your business license or other means of verification that isn't a bad idea. As an example people paying with POs today are excluded from the algorithm already.

Yeah... that one was painful and we are fixing it. At least if the priority placed this at the top of the queue we could have acted faster. Probably the same outcome due to the other issues involved in this incident though.

I was very worried about that specific detail and we reached out to the customer before posting this postmortem to expressly get his permission to share those details. If he had said no, we would have worked around the detail but not be able to explain as clearly what went wrong. He gave us his permission to share the information.

If you want to do cryptocurrency mining on DO that is actually okay with us. Some of the other respondents are correct the behavior we were looking for was really around fraudulent accounts being created and performing cryptocurrency mining. This is why the trigger that flagged this account was using payment history as a key factor in the triggering.

Last week ended on a real low note for many of us at DO. We took a perfectly good customer and gave them an experience no one should have to go through (all while he was trying to leave on vacation no less). We can and must do better. To do better we need to learn from our mistakes. To that end, we also think sharing the information about this incident openly is the best way to help all our customers understand what happened and what we are doing to prevent it in the future.

Yesterday we completed our postmortem analysis of the incident involving Nicolas (@w3Nicolas) and his company Raisup (@raisupcom). With their permission we are sharing the full report on our blog here:

https://blog.digitalocean.com/an-update-on-last-weeks-custom...

Yup. LeonM, you are correct. In this case that was the cryptocurrency mining detector that was triggered. More details in the postmortem.

I wanted to provide you all with an update on the postmortem I promised on Friday. Our analysis has been completed. We will be sharing the full document soon and will publish a link in this thread for those wanting to read it. We promised Raisup a first look and we have provided the draft document to them this afternoon. Because some information in the document could be considered sensitive we wanted to give Raisup a chance to review the document before sharing with the public.

Thanks for the replies. Let me try to address a few of the things I have seen here. We haven't completed our investigation yet which will include details on the timeline, decisions made by our systems, our people, and our plans to address where we fell short. That said, I want to provide some information now rather than waiting for our full post-mortem analysis. A combination of factors, not just the usage patterns, led to the initial flag. We recognize and embrace our customers ability to spin up highly variable workloads, which would normally not lead to any issues. Clearly we messed up in this case.

Additionally, the steps taken in our response to the false positive did not follow our typical process. As part of our investigation, we are looking into our process and how we responded so we can improve upon this moving forward.

As DigitalOcean's CTO, I'm very sorry for this situation and how it was handled. The account is now fully restored and we are doing an investigation of the incident. We are planning to post a public postmortem to provide full transparency for our customers and the community.

This situation occurred due to false positives triggered by our internal fraud and abuse systems. While these situations are rare, they do happen, and we take every effort to get customers back online as quickly as possible. In this particular scenario, we were slow to respond and had missteps in handling the false positive. This led the user to be locked out for an extended period of time. We apologize for our mistake and will share more details in our public postmortem.