CrowdStrike to Delta: Stop pointing at us
wsj.com
wsj.com
@zug-zug wrote:
> While this is technically what crashed machines it isn't the worst part.
> CS Falcon has a way to control the staging of updates across your environment. businesses who don't want to go out of business have a N-1 or greater staging policy and only test systems get the latest updates immediately. My work for example has a test group at N staging, a small group of noncritical systems at N-1, and the rest of our computers at N-2.
> This broken update IGNORED our staging policies and went to ALL machine at the same time. CS informed us after our business was brought down that this is by design and some updates bypass policies.
> So in the end, CS caused untold millions of dollars in damages not just because they pushed a bad update, but because they pushed an update that ignored their customers' staging policies which would have prevented this type of widespread damage. Unbelievable.
Link to video:
> Now, as to the tidbit. Dave Plummer ran a scam company that was sued by Washington State in 2006, "SoftwareOnline.com, Inc. ". He actually left Microsoft specifically to run this company.
> Court documents can be seen here: https://www.atg.wa.gov/news/news-releases/attorney-general-s... You can find David W. Plummer listed in the court complaint.
> The short of it is that it was an online software scam company that tricked people into downloading fake Anti-virus and security software using online ads, and then the software delivered additional adware and nagware onto users machines.
That doesn't invalidate the parent comment tough
Their staged update process is for the falcon driver itself. It is not for the "channel files".
As I understand it, the driver itself is understood to be a risk, and they provide facility for an N, N-1, N-2 staged deployment to mitigate this risk.
As I understand it, channel files were not identified as a risk, and were never subject to this staged deployment.
The "sell" was that you could be running a trusted driver at N-2, but still have 0day protection from up-to-date channel files. And CS's initial feedback that the issue was not with the driver itself was CYA that they hadn't been misleading customers using such staged deployments.
I did suggest we turn off the proxy for the "air gapped" parts of the nextwork, and only turn it on when we're sure we're ready for it so the airgapped parts can get the updates they need. but seriously... since when is it acceptable to give a vendor control that YOU DONT HAVE over parts of your network.. crazy days.
I've been at companies where we would do a bunch of processing overnight. One time, CPU died on a server in the middle of processing. Apparently we didn't have procedure for recovering in the middle of a process. Took us two days to write code to recover and we had to contact a third party vendor to get "What data did we send you?"
Companies handling critical infrastructure should face more scrutiny imo.
The reality is the industry wants its cake and eat it too. No one forced Delta to buy a software which could force upgrades in their production fleet. They're a billion dollars company, and should put their big boys pants on.
In my mind, a quick test run of the update on a VM before letting it roll out globally would have revealed the BSOD boot loop.
Crowdstrike should have done a better job, but Delta chose them (to offload the responsibility and work) and now they're claiming foul. They knew the risk. This is a classic executive play of claiming the fault lies in the consultants/vendor and taking no responsibility.
I guess you're saying they shouldn't have outsourced in the first place? Which does sound like the correct conclusion in this case...
lol.
Except this update was one from CrowdStrike that would ignore Delta's stated update policy.
And they literally said "Oh, yeah, we can configure some updates to bypass your policy". I wonder how well this was communicated to those customers.
The update policy may work for the client version updates, but not for the "policy definition", otherwise delta won't get the sweet "all vulns mitigated with a 4h SLA" they crave.
Well, that's nice. If I understand correctly, though, you can't contractually limit liability for gross negligence. I mean, you can say it in the contract, but it isn't legally enforceable.
It does raise the bar, though - gross negligence is harder to prove than ordinary negligence.
Note well: IANAL. I could be wrong.
Talk about a strategic vulnerability. "It's for security" they said.
Now imagine that a nefarious actor managed to compromise the FSA and push a stealth root kit. I shudder to think about how a full-out cyberwar would go.
Maybe Microsoft encouraged Delta to use Crowdstrike, then ok I could see the case.
But if it is anything related to how Windows works, that seems like a stretch. Yes Windows could be better, but I don't like the idea that the OS provider could be sued for a piece of software causing problems. That seems... dangerous.
Regarding Delta. This seems like an interesting situation. Apparently Crowdstrike offered people to help, but how many people and where. Realistically how much help would they have been able to help given that it isn't like this was isolated to Delta and I imagine their people were stretched pretty thin. And importantly what exactly was the help provided. I don't see any information about this.
While I am sure that Delta's IT department was understaffed, this was also a unique situation. If you spent the time to make a well optimized machine for rolling out updates, things were automated, and you expected things to go wrong but I would have never anticipated every Windows machine being unable to boot. That is an extra-ordinary situation. I doubt any IT department is really staffed to be able to handle that situation happening. You don't expect to need to deal with every machine you have, its honestly kinda unrealistic.
It will be interesting to watch this one play out in the courts. Because at the end of the day the vast majority of the blame for this entire situation is on Crowdstrike. The companies were handed a situation that we were not prepared for.
I do not believe this will be the last of its nature in our generation. It should very much be already have been accounted for. The basics and fundamentals of information security apply despite what interpretations of insurance policies or certifications say. Let this be a lesson to not introduce single points of failure into critical systems without having prepared for their unavailaility or misbehaviour (yes, that includes your ISPs, cloud, and SaaS providers
I think that is where this is going to get more complicated, I think that this has broadened what we traditionally think about when it comes to what we consider a single point of failure.
I would wager that most people would not have considered something like Crowdstrike to be a single point of failure. It is not what you would generally think of.
From what sounds of the bits and pieces of Delta that came out, it sounds like the biggest issue they had was that so many of these systems were not together. It required physical access to systems that were in many different locations (I would love to be corrected on this). Under normal circumstances that would cover you in the case of an outage in a particular location. If you built your system to handle entire areas going down, adding in redundancy.
But this was very much a unique situation. We really shouldn't pretend otherwise. It wasn't an external service going down, or AWS going down, loss of internet connection, or other things going down that we would normally account for in disaster recovery. Hell I would argue that in this particular situation, you could have the best disaster recovery but it wouldn't have done anything since Crowdstrike was probably baked into your images. So have fun bringing up an instance that was going to instantly brick. Yeah eventually that would have been a non issue, but I can not imagine a scenario that I would have ever thought that I had a baked image and assuming I still had access to it and I could assure that it was the image that I made, that it somehow would itself be a problem.
Before a couple weeks ago if we had built a system that was redundant enough to handle parts going down, different geographical locations, etc etc with, maybe things slow down but still works. All of the standard things that we talk about.
It isn't absurd to think that your entire system going down in a situation where you don't think you have a single point of failure, is nearly impossible or if it happens there is something outside of your company seriously bad.
Sure. Most people are not decision-makers for critical IT infrastructure. Should we similarly throw our hands in the air if a bridge or building collapses because the people responsible didn't pay attention to structural safety just because it wasn't obvious to a layperson?
> Hell I would argue that in this particular situation, you could have the best disaster recovery but it wouldn't have done anything since Crowdstrike was probably baked into your images.
It's not really "the best disaster recovery" then, is it?
> Yeah eventually that would have been a non issue, but I can not imagine a scenario that I would have ever thought that I had a baked image and assuming I still had access to it and I could assure that it was the image that I made, that it somehow would itself be a problem.
There are so many other ways this can back-fire I'm not going to try to enumerate them. I really hope no one put you in charge of operating anything critical...
Which may be a fair statement, but I think it's also fair to litigate whether or not this post-incident statement is consistent with how CrowdStrike sold their software to Delta.
Software fails, machines fail, we all know this. The technical leadership at Delta should know this. Do we think software vendors should be responsible for ensuring their customers safely deploy their software? I can't imagine that playing out well.
When it's priced differently depending on how the customer plans to use it, and it's sold only through a consultative process, I'm not sure it's so clear. In order to determine their pricing, the sales team demands to know how the customer is using it... so they absolutely arguably do know.
I should be able to do whatever I want with a computer I buy, but that doesn't mean Microsoft should hold any liability for it. What am I missing here?
If Microsoft's SLAs are worded so that you're allowed to shove bits of third party code into the kernel without violating the SLA, though, they should fix that.
This just sounds like shifting the blame to me. Nothing Microsoft did changed anything for the CS situation. Apparently it's Microsoft's fault that they don't have a great, secure, bugless API that everyone seems to want all of the sudden.
On the day the crashes started, Microsoft had a cloud services outage, and people assumed the two outages were related (they sort of were, in that some of the consultancies Microsoft hired also ran CrowdStrike so they couldn't get support for the cloud outages as easily). That seemed to have stuck around.
I think their lawyers are trying hard to convince Delta to not sue them and they will pay in back channels. As long as it's obstructes from public news, it's a win for them.
Am I reading this right that this amounts to "you decided yourself to install our software on your devices, you should have known better?"
> “Should Delta pursue this path, Delta will have to explain to the public, its shareholders, and ultimately a jury why CrowdStrike took responsibility for its actions—swiftly, transparently, and constructively—while Delta did not,” wrote Michael Carlinsky, an attorney at law firm Quinn Emanuel Urquhart & Sullivan.
or maybe it's just how Delta didn't take their offer for help
> The cybersecurity company reiterated its apology to Delta for the initial disruption and said it had offered on-site assistance to Delta but was told it wasn’t needed. It said Bastian didn’t respond to outreach from CrowdStrike’s CEO.
> CrowdStrike said Sunday that its liability is contractually capped at an amount in the “single-digit millions.”
That's probably a key question too^
So lawyers trying to pawn this off and use stupid legalese to shift blame, make or ask time-wasting accusations etc. I'm not a lawyer or versed in all the legal stuff, but something smells like a double standard here.
This is certainly a learning experience for MSFT and CS, but I think it's foolish to say they are at fault. Delta has a choice in what software they employ.
Delta was responsible for the rest of its outage and should deal with any damage that it caused.
edit: delta should have had better system outage processes