CrowdStrike says it isn't to blame for Delta's flight cancellations after outage
cnbc.com
cnbc.com
if CrowdStrike isn't to blame, then who is?
Even if we accept that CrowdStrike’s fuckup was the root cause of all the issues that Delta customers experienced, the level of impact CS’s fuckup was able to have is entirely on Delta. Their choice of platform, their level of resilience, the processes they have in place for rapid recovery, etc, are all Delta’s responsibility.
There is zero complexity here. CrowdStrike accepted money for security services and offered a client. Then they didn't bother to test their stuff and literally caused the biggest IT outage in history to save a few bucks. They literally have a CEO with a history of not caring.
If you offer your stuff for use in critical infrastructure you cannot do stuff like that. This company needs to be sued into bankruptcy. This was plain bad engineering. If the same happened with a bridge, and it collapsed no one would even hesitate to lay blame on the engineering company/architects. You have a choice if you want to drive over a bridge or not, but when it collapses and you are on there I doubt you'll think I have to accept the consequences of my choices on your way down
Perhaps Delta had a shittier recovery plan.
> Perhaps Delta had a shittier recovery plan.
Or Delta has shittier software. News articles kept mentioning Delta's crew scheduling system couldn't keep up with its backlog after it was brought back online, and I take that to be the reason they had to cancel flights for so long.
If true, that's not Crowdstrike's fault, it's Delta's for not investing enough in their own mission-critical systems to make them resilient after an outage.
I don't think it's unreasonable to hold a company with $60 billion annual revenue, and millions of customers to be affected by any faults, more responsible for inspecting the infrastructure they choose and having contingency plans in place than an individual with little means to realistically do that. Particuarly when, as you note, there was a history of not caring to save a few bucks.
If not CrowdStrike, it seems as though something like a ransomware attack could've caused the same situation.
Delta are responsible for not having good disaster recovery systems, and for purchasing CrowdStrike's software.
It's worth checking for any warranty disclaimers, particularly disclaimers of the implied warranty of fitness for a particular purpose. If they exist, the software shouldn't be purchased. While the licenses to enterprises are private, their website's license explicitly disclaims such things as them installing viruses on your computer:
> AS BETWEEN YOU AND CROWDSTRIKE, YOUR USE OF THE WEBSITE IS AT YOUR OWN RISK. THE WEBSITE IS PROVIDED TO YOU BY CROWDSTRIKE ON AN "AS IS" AND "AS AVAILABLE" BASIS, WITHOUT ANY WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED. NEITHER CROWDSTRIKE NOR ANY PERSON ASSOCIATED WITH CROWDSTRIKE MAKES ANY WARRANTY OR REPRESENTATION WITH RESPECT TO THE COMPLETENESS, SECURITY, RELIABILITY, QUALITY, SUITABILITY, ACCURACY OR AVAILABILITY OF THE WEBSITE. WITHOUT LIMITING THE FOREGOING, NEITHER CROWDSTRIKE NOR ANYONE ASSOCIATED WITH CROWDSTRIKE REPRESENTS OR WARRANTS THAT THE WEBSITE, INCLUDING THE WEBSITE CONTENT OR ANY PRODUCTS, SERVICES OR MATERIALS OBTAINED THROUGH THE WEBSITE, WILL BE ACCURATE, RELIABLE, ERROR‑FREE OR UNINTERRUPTED, THAT DEFECTS WILL BE CORRECTED, THAT OUR SITE OR THE SERVER THAT MAKES IT AVAILABLE ARE FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS OR THAT THE WEBSITE OR ANY SERVICES OR ITEMS OBTAINED THROUGH THE WEBSITE WILL OTHERWISE MEET YOUR NEEDS OR EXPECTATIONS.
If your AV software's publicly facing website needs to disclaim liability for "viruses or other harmful components" that their site may serve up, you really ought to think twice about giving them money. After all, if it were a useful anti-virus then they wouldn't need to worry about people getting viruses from their own site, they'd just use their own product.
Well, if you as a customer decide to install said trojan software on all your IT equipments, you should be ready to have to recover all those IT equipment at any point in time. That is that simple. If you can't do that in a reasonnable time, that is on you as well.
Yet in software this expectation is always absent.
To a certain extent I get it, although the root cause was the CrowdStrike issue, the knock-on effects were exacerbated by Delta’s duck-tape-and-prayers existing IT services.
Makes it harder to accept responsibility for that when you’re being sued.
Delta took 6 days to recover from something as impactful as this, which is much worse than any other airline. So much so, the US Transportation Department is currently reviewing why it took so long.
It is vital that any critical infrastructure such as this has a low MTTR, which Delta clearly failed to plan and test for, therefore the IMPACT that Delta felt cannot be solely attributed to CS, albeit they did trigger it, but Delta failed to mitigate the impact.
Otherwise they were forced to NOT lock down the kernel at Vista time.
Imagine if your bios vendor made it trivial for an application to brick your hardware. Yes, you'd blame the software, but you'd also be asking why the bios didn't protect the system.
In Linux eBPF is a sane interface which CS now use. MS are now developing their own eBFP interface, but it's only a early concept.
According to former Windows developer David Plummer, Microsoft does, in fact, offer a number of APIs for third-party antivirus security. “CrowdStrike defaults to kernel mode, presumably because it needs to do things that can’t be done from user mode,” Plummer said in a YouTube video.
“And to me, that’s where Microsoft could be responsible, because on the Windows platform, to the best of my knowledge, some of the CrowdStrike security functionality requires deep integration with the operating system that can only be currently achieved on the kernel side.”
Microsoft has a number of APIs including Windows Defender Application Control API and the Windows Defender Device Guard, which Plummer said provide mechanisms for controlling application execution and ensuring that only trusted code runs on the operating system.
He said that the Windows Filtering Platform (WFP) allows applications to interact with the network stack without requiring kernel level code. However, quoting sources within Microsoft, Plummer claimed that the company had actually “tried to do the right thing” by developing an advanced API designed specifically for security applications such as that from CrowdStrike.
“This API promised deeper integration with the Windows operating system, offering enhanced stability, performance and security,” he added.
But the EU 2009 ruling effectively prevented such integration as it could potentially have given Microsoft an unfair advantage.
However, Ian Brown, an independent consultant on internet regulation, argued that Microsoft should have better security controls, rather than attempting to put the blame of the CrowdStrike crash on the EU anti-competition commission.
In a blog, he wrote: “For technology-dependent societies’ resilience, OS kernel-level software and equivalents on socially critical infrastructure systems (like travel, healthcare and banking) need to be very carefully tested (and ideally run on top of a formally verified microkernel) and controlled. But OS monopolists shouldn’t be making the final decisions about precisely what those controls look like, where they have implications for competition.”
If you give third parties access to your systems with proprietary code without any checks or balances because you are too lazy to fight for the budget to deploy an in-house alternative, then whatever bad results happen are partly on you.
Yes, sue CrowdStrike, but also fire every sysadmin and IT manager that recommended or approved installing it on systems in the first place.
No one acting alone can deploy a change to mission critical systems I design.
This is just basic supply chain integrity.
Asking what the falcon agent does, how it is updated, what it has access to, and if it can be remotely controlled is part of the job of any IT engineer / manager who plans to purchase a license for crowdstrike. If you don't, you aren't doing your job.
I guess it is your fault, because you failed to do the allergic chemical test, before eating it.
CrowdStrike was clearly responsible for some of it. It’s pretty clear that Delta had other major issues that were also triggered by the CrowdStrike outage which it wasnt responsible for since other airlines didn’t face those issues.
You raise a valid point, but imo the key question (to which I don't claim to have an answer) is whether Delta's extended problems are due more to their disorganization or simply to bad luck.
Here's a somewhat silly analogy: Imagine Kellogg's introduces a new ingredient in their cereals that triggers an adverse reaction only in people who have eaten sardines within the last hour. It would still be correct to hold Kellogg's entirely responsible for the adverse reactions experienced by those individuals. It wouldn't be a valid defense for Kellogg's to say, "Most people didn't experience any issues, so it can't be our product causing your adverse reactions".
Sandbox rules simply don’t apply when real money is at stake— the contracts that sit behind these relationships are all that matters + a companies ability to stop doing business with one another.
Delta probably isn’t even entitied to a pro-rated refund of their prepaid CrowdStrike subscription. If Delta has a multi-year deal contract with CrowdStrike, Delta most likely have to keep paying CS for some time In the future.
CrowdStrike breached but almost certainly cured within allowable period.
Maybe they sue for gross negligence which I think may circumvent contractual liability limits in certain situations.
If push comes to shove, Delta can sue and/or stop using the product.
This is ultimately a question of contracts, liability limits— particularly if Delta secured consequential damages.
SaaS contracts are designed to defaulted to NOT allow a customer to pursue consequential damages remedies.
https://en.m.wikipedia.org/wiki/Consequential_damages
This is a question of CrowdStrike’s Deal Desk contracting hygiene.
Deal Desks are the joint finance-legal-sales teams that work on enterprise contracts in scaled enterprise SaaS startups.
This is a SaaS CFOs nightmare.
But Delta for having terrible investment in modernizing their IT infrastructure.
This is very likely to settle out of court or dropped once the CS outage falls out of the news cycle.
Should Delta pursue this path, Delta will have to explain.. why CrowdStrike took responsibility for its actions—swiftly, transparently, and constructively while Delta did not.. Delta would have to preserve a series of documents, including those describing its information-technology infrastructure, IT business continuity plans and its handling of outages in the past five years
If only some smart tech people gathered somewhere and someone could make a mobile app to allow crew to set their status and location instantly. They’d corner the market and save airlines billions.
Society still blames the match based on recent legal outcomes, so Delta will probably win the argument.
It was no secret CrowdStrike updated all PCs at once. Delta could see that. It is no secret updates can nuke computers. Delta knew this could happen.
All business should have a plan for cascading/total outages.
This was a great test of humanity that Delta seems to have failed.
Deltas total incompetence makes me feel like CrowdStrike incompetence is canceled out. Delta's CEO clearly has no idea what he is talking about.