Thousands of Delta passengers delayed by computer outage | Hacker News Reader

Thousands of Delta passengers delayed by computer outage

bbc.co.uk

115 points0xbadf00d121 comments

The most annoying part of bugs like this, is that we rarely get a good post-mortem. To frequently the media just reports it as "a computer glitch". You would think that with computers being so important to modern day life, we should get better answers.

I'm not suggestion that the bug should be explained in details in the evening news, but at least give some indication of the nature of the problem. They could just refer to a website, if people want the technical details.

When a car has a problem, it's not reported as a "glitch". Imagine the stupidity of a media report claiming "A glitch in the latests Ford models cause several drivers to crash". You don't do that, you explain that it's a fault in the steering, breaks, electric systems, or where ever the issue lies. I think we should reasonably expect the same when errors in computer systems hit the news.

I'm afraid that's because the target audience has (on average) that level of technical knowledge (in both cases)...

One of the main reasons that you don't get a good post mortem is that it's often directly related to human error or a severe defect brought about by a decision to cut costs. There have been several huge outages in UK banking over the past few years and management have a convenient excuse in blaming "legacy systems". While I respect private enterprise, for systems of systemic importance I would like to see some legislation to implement a mandatory and detailed post mortem like you propose.

>it's often directly related to human error or a severe defect brought about by a decision to cut costs

I wonder if companies do cost-benefit analysis on those kinds of situation. "We saved X amount of £ on cutting costs. Outages as a result of cost cutting Y amount of £". You would think that at least stock holders would insist.

No because doing that would mean knowing what happened and spending time doing a post mortem.

This may come as a shock but most people want to do a good job, even if they work for Citi or Delta or Comcast or where ever.

Okay, maybe not Comcast.

I never said the contrary. But a tons of these systems are old, and people knowing how they work are long gone.

Plus doing a proper post mortem and understanding how it works woudl take a lot of time and money. Things they most of the time to not have budget for.

In a past life I was an IT consultant and good organizations do ask this question. Sometimes you have to ask the question like that to get business leaders (or board members) to pay attention to that "new IT spending".

I love doing that =D

A while ago (when I was in aerospace) there was one senior people who gave us the price of insurance per head.

Given that we're talking about 1M€ per head and that a plane can easily have 200 people. I have a hard time finding a situation where I can expect a Return On Investment by doing things poorly and cheaply. We're talking many millions of payout.

(And yet, I'd love to have such a situation, it's always good to attract the attention of attendees at conference or student classrooms).

Then later I realized that I've made a mistake in my calculation... I'm only accounting for A SINGLE plane failing and killing everyone. In the real world, a bad component would be shipped to a whole batch of planes and they'd fall randomly like flies.

That batch-effect drastically increases the cost of failure. So... I really don't see ANY place where there is a ROI on people dying.

BAZINGA!!! I'm from Europe and people are expensive there. The trick is to manufacture things for the 3rd world market where people get peanuts when they die (sometimes even nothing) :D

But then again... we're back to square one, how could I build parts AND guarantee that they're only shipped to planes in low-cost locations? At that point, it's probably closer to a plane-by-plane targeted sabotage rather than a scheme of doing minimum-shit-to-close-sales. And when doing plane-by-plane there is no economy of scale :(

... Anyway! Just wanted to say that any analysis would clearly reveal that THE PLANE IS CHEAPER WHEN IT FLIES!!! (and your life is truly expensive) so you're fine. You can keep on using airplanes ;)

Also, people at work do have a conscience. Once in a while they may forget about the importance of what they do, luckily there is a plane crash happening every few months to remind them that there are lifes at stake, for real (in case it was ever forgotten).

Another big reason is security - the less a nefarious actor knows about a system, the better (at least from the perspective of a large, closed organization).

Now we know this is not always true, but even giving out details about schema, data-organization or technologies used could provide more detail than is necessary.

We could argue the merits of this approach - I can certainly sympathize with the idea - but security by obscurity is still a rampant ethos in huge corporations. I wouldn't be surprised if TSA also enforced a bit of this as well.

This makes sense in a vacuum, but it is pretty easy to figure out any decent sized company's tech stack just by browsing their IT group's LinkedIn profiles.

Good postmortems should be created for human-induced errors or top-down decision to cut costs. Especially for those.

You don't need a post-mortem when you already know what the problem was.

If you knew what the problem was before the outage happened, and didn't consciously put the effort to prevent it, then oh well, that's professional negligence. Maybe not on you because you may not have the decision power to apply large-scale changes, but definitely somebody within the company. And if you didn't know what caused the outage, and you conducted some type of investigation to get at the heart of the problem, then you need a postmortem.

That's one of the main goals of a postmortem: document what the cause of the problem was, not just for you, but for all interested parties. If some human caused the issue, you need to document what happened and what should be done to prevent it from re-occurring. And fix it, of course.

Can't we all just assume from now on that anything bad that happens is because of some shitty CEO cost-cutting everything to line their pockets? Because 99% of the time that's what's happening.

Delta being down for four hours is not an issue of "systemic importance". Safety-related issues are already (and rightfully) publicly dissected, but it's hard to see what a post-mortem showing (what is likely to be) the interplay of a couple of subtle and individually benign problems which when hit by the power outage became a big issue will do much except enable a large chorus of armchair experts to pronounce on how this is obviously the fault of evil cost-cutting management/because they didn't use the cloud/because they used the cloud/fixed in the latest version of node.js.

I did not say that Delta being down was of "systemic importance". However a bank being unable to send or receive payments for a week is of systemic importance; hence why I would like a regulator to decide what is of systemic importance e.g. banking, utilities, telecoms. We are getting more and more where broadband is out, banking is out, etc. and the root cause is never transparent.

If a bank's systems are down consistently, use a different bank. Where public safety is not a concern, it is not any of your business as a private citizen what the inner workings of a private enterprise are, including the cause of outages even if they affected you directly.

That being said, of course I am much more likely to patronize businesses who do give me this type of information. Given a choice between a bank with satisfactory service and amazing transparency, and a bank with great service and no transparency, I will choose the former every time.

But government regulators opening up the working of private businesses is not a proportional response to your bill pay being unavailable for a few days.

A ticketing system may not be "safety" critical, but its malfunction can cause tremendous discomfort and financial strain on people's lives. Being stranded far from home, missing an important event, or being put through hours of incredibly stressful situations is inexcusable.

Then don't ever fly Delta. Consider this your official notice.

You look at it from a narrow consumer perspective. If a bank's IT systems fail for a period then the systemic importance clause kicks in, as the wider economy may be affected. Regulation already exists e.g. You can't fix interest rates, you have to be fair to consumers, etc. In today's modern world I want regulation on how some of these enterprises are managing the technology which underpins their operations.

Financial outages get punished by investors anyway, and they all want to keep their competitors from knowing how they operate, so I'm not sure there is a strong interest in getting this right.

In the case of airlines though, they have a strong collective interest in avoiding incidents.

This will likely never happen since it puts the offending company in the position of having to publicly air their internal problems, whether it's a lack of proper power backup, inadequate staff, etc.

Most people know what steering, breaks, or car suspensions are and a large chunk of the general population has a mental model of how they work.

After the power outage of 2003, the NYISO released a post-mortem [1], but I don't remember CNN reporting on it. This is probably because it takes a significant quantity of effort to turn a detailed report into a 10-minute video.

I imagine that a youtube channel that ran post-mortems on large-scale systems would be popular.

[1] http://www.nyiso.com/public/webdocs/media_room/press_release...

Does anyone know if a good resource of detailed postmortems even if really old? I love reading these but they are often way too short on details.

https://github.com/danluu/post-mortems

Awesome. Thanks.

Technology seems to be a black box for journalists, including technology journalists. I'm expecting them to start using words like 'boffin', in addition to 'glitch' to show that they have given up trying. That's the language of tabloids (and outlets that do a tip-of-the-hat to tabloids).

Marginally better: The official post cites a power outage in Atlanta.

> A power outage in Atlanta, which began at approximately 2:30 a.m. ET, has impacted Delta computer systems and operations worldwide, resulting in flight delays. Large-scale cancellations are expected today. All flights en route are operating normally. We are aware that flight status systems, including airport screens, are incorrectly showing flights on time. We apologize to customers who are affected by this issue, and our teams are working to resolve the problem as quickly as possible. Updates will be available on news.delta.com.

http://news.delta.com/more-flights-resume-delays-cancels-con...

It's true that the power outage was technically in Atlanta (because that's where their data center is), but it's misleading because it was only their data center that went down. The utility company claimed no other customers had outages.

This is nearly a completely different story. Ace reporting.

By way of contrast, the National Transportation Safety Board does a pretty thorough investigation of aircraft crashes and records results in a public database [0]. The software industry has, so far, managed to avoid this level of scrutiny and liability. The jaded part of me says that it will take a crisis to get software to be taken that seriously, but the crises so far haven't been large enough.

[0] http://www.ntsb.gov/_layouts/ntsb.aviation/index.aspx

This is a really interesting point. I think when software starts killing people 100 at a time (like a plane crash) we'll get that kind of oversight.

As the other poster indicates, right now it's more like software wounds by the thousands or millions, and, so far, that's not been enough.

There have already been crises, it is just that some crises are more subtle (though still quite devastating.) Consider the crisis of the Washington Mutual post-merger system integration. They purchased dozens of banks in the hopes of becoming a super-bank, but the details after the deal closes are often neglected. I was a Dime Savings Bank customer. My beneficiaries (upon death) on the account magically got converted into account holders, though only in the 1099 system, not on the reporting/statement system. We didn't even know until the next year when my brother's student aid was declined -- they noted he had ~$17,000 in his bank account, which was curiously about how much I had. He didn't even have a bank account!

Something like this can literally kill a semester/year of college if you arent savvy enough to deal with it. Luckily for us, tens of thousands of other customers had the same issue and the school was inundated with similar cases -- they realized something was amiss.

Incidentally, Washington Mutual refused to help -- they noted that the statement did not show my brother's name. They also noted that they cant talk to my brother since he isnt a customer!!!

I'm so sorry for your troubles. I think there's a huge price individuals are paying for these kinds of things. A much easier case than yours, I once spent months trying to convince a phone company that they'd applied my payment to the wrong account. It took a letter to my state's Public Service Commission to get them to pay enough attention to solve it.

Its usually a complicated interaction involving old systems, new systems, unusual circumstances and an obscure bug. Delta Airlines was started in 1924, so its a fair bet they've been using computers since at least the 1970s. Environments like that tend to accumulate legacy systems, because every time some new requirement comes along (teletext boxes in travel agents? the Web? Passing data to Homeland Security?) its easier to add a new box on the side instead of re-architecting the existing legacy. So the result, after 4 or 5 decades, is not like a "latest Ford model", its more like Johnny Cash's Cadillac in http://www.metrolyrics.com/one-piece-at-a-time-lyrics-johnny.... Want to bet that car was stable during an emergency stop in wet conditions?

Its easy for us on the outside to say "well, have a backup" or "just replace all the legacy systems", but in the real world its not that simple.

Yes, you can have a backup server, but it has to be running exactly the same software and working on a copy of the data. If some unforeseen event corrupts the database, its likely to corrupt the backup replicant in exactly the same way. Also its likely that some of the legacy systems are not designed to support replication and can't have it retrofitted with any reasonable degree of cost and risk.

Replacing a single legacy system with another system that does exactly the same thing is expensive, risky, and wins you almost nothing. You have to replace a large chunk of the whole legacy all at once, without disrupting your day to day business. This is hard to do; any project to try this is likely to fail before it really gets started.

The way to do this is to create an automatic translation tool. Furthermore, this translation tool can't be generic. Instead, it has to map the idioms of the legacy project to new code in the idioms of the new environment. Done properly, doing the port at one level of indirection like this, you can absorb all of the changes to the legacy system as they are happening, then instantly switch over when the new system is finally ready.

Got any case studies where that worked?

There was a consulting shop that did nothing but this for Smalltalk to Java. I also did such a project. Yes, this works, given good parser tools with scripting and full syntactic expressiveness.

It's not software or hardware this time: there's a crippling power outage at ATL, which is Delta's main hub (and presumably where their mainframe herd is based):

http://fox61.com/2016/08/08/delta-airline-reports-outage-eve...

The outage began at 0230 and is presumably ongoing and beyond the capacity of their backup generators to cope with.

Same here:

http://news.delta.com/730-am-et-update-outage-affects-depart...

A power outage in Atlanta, which began at approximately 2:30 a.m. ET, has impacted Delta computer systems and operations worldwide, resulting in flight delays

8:40 a.m. ET UPDATE: A Delta ground stop has been lifted and limited departures are resuming following a power outage in Atlanta that impacted Delta computer systems and operations worldwide. Cancellations and delays continue.

I'm quite surprised that their whole operation is region dependent. I work with hospitals on high availability/continuous availability of critical systems and would think that a company as big as Delta would consider a regional power outage as a high risk and plausible scenario.

Have you flown through Hartsfield-Jackson? There's six concourses with 30+ gates each, nearly all of which is Delta. Their operations are highly-concentrated there.

That being said, I expect them to upgrade their generators to handle outages like this, and distribute more of their IT infrastructure. Unlike the typical summer storms in the Atlanta area that delay flights, this affected them worldwide.

As a passenger on the west coast flying to another west coast city, I give no shits about the number of concourses in ATL. LAX still has electricity but delta's poor infrastructure planning has left a single point of failure in the deep south affecting flights everywhere in the world.

So - too big to fail? ;)

LAX still has electricity but delta's poor infrastructure planning has left a single point of failure in the deep south affecting flights everywhere in the world.

The "all over the world" bit is why there's a single point of failure.

A while back there was a story that made the rounds of aviation geeks, about Delta flying an empty 747 to Korea. That was to replace a 747 which had been badly damaged by hail, to the point that it would be unable to operate its scheduled return flight to the US.

Do you want to guess how many people, parts and places were involved in the "simple" task of dealing with this problem?

At first, the replacement 747 is in storage in a "boneyard" facility in Arizona, due to having been recently retired. So first it has to be pulled from the storage facility, put through basic airworthiness checks and fueled up, and then a flight crew has to be present to fly it to a Delta hub where it can be readied for a trans-Pacific flight.

The hub in question turned out to be Minneapolis. There, the plane has to undergo more work to get it ready for a long flight, and now multiple flight crews have to be present, since they need to rotate in and out over the duration of the flight (that's how you do long flights). Oh, and Minneapolis isn't normally a 747 base; it only gets them during peak travel seasons and on the occasional charter. So crews probably have to be brought in, stores and maintenance setups need to be brought online, etc.

Then the plane can -- finally -- fly out to replace its damaged counterpart, pick up any stranded passengers and bring them to the US. Which will mean flying into yet another hub, since the flight doesn't go back to Minneapolis.

Meanwhile the damaged plane is still sitting there in Korea, and needs to be repaired on-site to get it into minimum airworthy condition to fly home (empty of passengers). It's going to need parts, maintenance crew, flight crews, etc. just like the replacement plane did.

And the deeper you dig the more stuff you'll find like this. Running an airline with global, or even national-across-the-US, service is not something you can decentralize to avoid problems at one operations center. The amount of coordination just of people, parts and planes across widely disparate locations requires centralized operational control instead of devolved regional centers with high autonomy.

Absolutely none of that is an excuse for having an entire airline dependent on a single data center. It's about redundancy. You centralize administration, not the control plane itself. Quorum in database systems, load balancers, and DNS updates solved these problems a long time ago.

At this point I consider a company as large as this having such a rudimentary single point of failure to be incompetence in the IT department. We wouldn't be so forgiving if delta needlessly kept all of its pilots in one city during the night so a single storm wiped out every flight.

You centralize administration, not the control plane itself.

You're still out of luck when the centralized admin center goes down, though. That's the place that is the source of all the humans performing the coordination and dispatching work. Having a bunch of extra data centers and backup generators around the country will not cause those humans to become accessible.

And building out full redundant continuity of everything, including the humans, is not something that tends to happen outside of major governments.

That's not what happened! It's the computer system that failed. The entire administrative team didn't just up and die.

Also, "centralize administration" just means that you can control everything from a single location. It doesn't preclude being able to control from multiple locations.

Think of AWS, you can control everything across multiple data centers from a centralized interface from anywhere with an Internet connection, even if entire data centers go down.

A sane system should essentially allow delta to operate from many possible locations seamlessly as long as they have the human operators required.

Subtle elitism is the best kind of elitism.

Right, you don't have to have your servers and IT infrastructure physically next your your gates to work.

I find that pretty odd. A decade and a half ago now I rented part of a rack in a not particularly high end co-lo facility. They had battery backup for an hour and then generators that could be fired up in twenty minutes with 3 days worth of diesel.

Delta airlines, with $40B of revenue last year, has less than 8 hours of backup power for their mission critical computers?

or even a second site to fail over to?

Exactly my question, why was there no failover? This outage is going to cost them substantially.

Even on classic DBs, setting up a failover with a lag of 15 minutes isn't a herculean effort. It's expensive, yes, but probably worth it...

We once had a long power outage (3.5 hrs ish), the data center remained on the entire time, but after about 60 minutes the data center lost internet access(!) because their ISP only had a battery backup with a limited capacity. So it does happen.

Plus sometimes backup power isn't reliable (e.g. generator problem) or they have different tiers of coverage (and Delta didn't pay for it). We'll just have to wait and see.

That's why anyone who actually cares about keeping their business running keeps a disaster recovery site in a different data center, and regularly tests their failover procedure.

It depends how much it would cost you to maintain vs the cost of a possible downtime.

There are things that are fairly easy to make redundant. I don't know anything about their infrastructure, but I doubt building a failover would be trivial.

Even if you had it in place, the risk of something occuring as you switch back and forth might not be worth it if you expect your regular setup to come back up within a reasonable time frame. Most people around here know how close to impossible it is to replicate real world usage on a test system, and how much overhead it adds to any change you make to your production system if you want the others to follow.

Both Equinix and AWS have had cases where the genset tests pass, but entire DC blackouts occurred due to a fault only a few months later.

Yup.

I've worked at several places where there has been battery backup, diesel generators, and a a disaster recovery site with generally nice and thorough strategies for replication including really expensive "sysplex-mode" for mainframes, topped with regular contigency tests where one half of the site is shut down.

And somehow, at every place, eventually something bad happened in a way that took down the applications.

It could be a power outage in the exactly the wrong circuit, fluctuating power messed up exactly the wrong storage system, someone dug up exactly the wrong fiber, the fire extinguisher set off by mistake and sprayed just the wrong cabinet with water fog through a door(!) that was supposed to automatically close, the diesel generator caught fire and the backup battery had to be taken offline, etc etc.

I'm not saying that you shouldn't try, but it's damn hard.

I worked for a company that thought it had a really top level power backup system. The server room had 15 minutes of UPS, all PCs had at least 5 mins. Out the back was a big generator with 24 hours worth of diesel. It was all controlled by a fancy control system that fed light displays in every room indicating if we were on mains power, UPS power or generator power.

One day during a power cut the generator kicked in as it was supposed to, but a minute later it spluttered to a halt. After much fiddling around by maintenance staff they gave up and we were all told to go home.

The postmortem revealed that the diesel had become contaminated with all sorts of gunk - water, grit/sand and lumps of unidentifiable "stuff". The best laid plans and all that.

Edit: And cigarette butts as well.

> They had battery backup for an hour and then generators that could be fired up in twenty minutes with 3 days worth of diesel.

How often did you test that?

> has less than 8 hours of backup power for their mission critical computers?

It's plausible that they thought they had that, but some part of it failed.

I don't remember, but when we were taking the tour the guide did mention that the tested the batteries and generators on some schedule. I bet there's some industry standard that has a minimum testing frequency.

In my experience (radio repeater and cell sites on mountain tops) an auto-start timer runs the generator once per week for 30 minutes or so. Diesels dont like to sit too long without being run. And these are 30-50 Kw generators, relatively small in comparison to a data center. I expect a weekly schedule is common practice.

> How often did you test that?

Not parent, but here is a single data point. When I was a student, I worked for some sysadmins who managed a small, internal data center. ~200 IBM servers, a SAN, an old VAX, etc. It had a backup gas generator that was tested loudly, every single week. It was also tested whenever the power went out for too long.

Obviously lots can go wrong, even with the best plans, but at least at the tiny little place I worked, the generator was tested every week.

The data center where my employer used to house stuff tested their generators for short bursts weekly, and would run them for 8 hours every quarter.

Very good points. Backups or backup systems should not be considered as true "backups" unless you've successfully failed over to them.

It can quite difficult to get a large diesel generator permit for a data center in the Atlanta area due in part to automotive emissions but also (ironically) the high greenhouse gas emissions caused by Hartsfield-Jackson Atlanta International Airport.

Mind you, I'm not saying they don't have one, just that it can be difficult to get -- there can also be limits on the number of hours you can run them.

What would stop you from just parking a large container outside the house? On top of a truck in the worst case... There are several providers of mobile power.

It looks like switching to the backup generator may have been what caused the fire that caused the outage in the first place.

> According to the flight captain of JFK-SLC this morning, a routine scheduled switch to the backup generator this morning at 2:30am caused a fire that destroyed both the backup and the primary. Firefighters took a while to extinguish the fire. Power is now back up and 400 out of the 500 servers rebooted, still waiting for the last 100 to have the whole system fully functional.

http://www.flyertalk.com/forum/27032000-post135.html

Update: Confirmed.

> But Georgia Power said the issue had to do with Delta's own equipment, not a larger power outage. "We believe Delta Air Lines experienced an equipment outage; other Georgia Power customers were not affected," said John Kraft, spokesperson for the utility. Georgia power has staff on site trying to assist Delta, he said.

https://www.washingtonpost.com/news/the-switch/wp/2016/08/08...

And of course, one of the benefits of diesel generators is that they aren't limited to that three days of fuel because they can be refilled while running. :)

I call BS.. Keeping the power on is a well managed process. I doubt a Data center like that will go down for power reasons. My bet is on ancient mainframes giving up. Been there, done that.

What are you basing this on? I've seen many datacenters fail to kickover to backup power for various reasons. This is a process that has to work in under like 30 seconds. For places with spinning mass, that's the amount of time that the spinning mass will power the systems while the generators are kicked on. [1]

The generators have been know to fail hard, and then the entire DC is down. On top of that, generators often are only able to run for a limited amount of time hours to days [2].

[1] https://en.wikipedia.org/wiki/Flywheel_energy_storage [2] http://www.graniteblockglobaldatacenter.com/powersystems.htm...

I wonder if Amazon would consider it an acceptable risk that a power outage in Seattle turns their stores off worldwide. Highly doubt it.

I'm sure a lot of people are asking how this is possible.

- Legacy systems: the thing is probably old and has had a lot of patchwork done to it by various people, many of whom are retired by now.

- The thing is also large and distributed, which makes noticing potential failures quite hard. There are loads of postmortems on the web about how some minor issue like a line going down caused some cascade of unforeseen problems. There's a lot of lessons being learned about how to solve this problem though, so I suppose there's hope.

- Always on: it's hard to change something once the business is relying on it. Loads of staff need to be shown how it works, which makes it hard to incrementally improve the thing because "how it works" would be something that people would have to be updated on.

- Never finished: large organisations will always have loads of feature requests and bug fixes. If you start responding to some of them, you might find yourself swamped. Either you work on the next item, or you spend valuable time making sure fewer items are fixed. It's a balancing act.

I would add Resourcing to the list. Have Delta airlines joined the mob of traditional big businesses who decide technology is expensive and send it to the cheapest possible?

Alternatively it could just be an catastrophic infrastructure failure.

Not likely. All of Delta's scheduling software is in straight assembly.

Deltas generally regarded as the well oiled machine of the airline industry. You're paying for always on time and consistent quality.

It's one of the reasons a story like this is surprising.

Interesting! Are there any source listings anywhere? Do they have particular programming tools or methods that help them deal with the complexity? If it's all assembly, I'm surprised this is only happening now.

Been that way for at least 2 decades. There's a big company in Atlanta that does all of it but I forget the name.

No source listings that I know of.

While I don't doubt their investment in their software stack to manage operations, Delta is hardly always on time nor consistent. I fly them 2-3 times a month and for the past two months (including today) every flight has has delayed (to and from destination) with only one being weather related.

Anecdotes from individual travelers are interesting but not very relevant or useful. The DoT publishes statistics for airline on time performance.

http://www.transtats.bts.gov/ot_delay/OT_DelayCause1.asp

Also hardware. Likely older and expensive mainframes, which while generally reliable do have a finite lifespan.

I'm guessing a critical hardware problem affecting some central part of their flight dispatching system.

Back in the 2000's Sabre purchased spare parts from the soviets to keep their ancient mainframes running.. I can't imagine how hard it must be now.

Not that hard if you're willing to part with lots of cash for IBM.

But that is mostly not true, you simply must invest money to replace and upgrade old system. Image similar excuses if airline was running WW2 era planes.

I saw a comment on reddit pointing out that planes cannot take off without the baggage weight/distribution calculation. And arriving flights cannot have their baggage scanned to show up properly at baggage claim.

The latter seems like something that could be overridden with pen/paper, but I can understand how the weight calc is intricately tied to the scheduling system. I just can't believe they can't have a fall-back at each airport: a simple program that runs on a PC that certifies a single flight based on who physically showed up and how heavy their bags are.

Power failure, clearly their back power systems are inadequate. I believe some of Delta's systems are also backed by SABRE which while ancient is generally rock solid (it lives in a nuclear hardened bunker in Tulsa). This problem is in Delta's own systems not SABRE.

the main host of delta is not sabre, it is deltamatic, a homebrew system that is run by delta. Most other airline systems like sabre communicate via teletype to one another

I think the reservation system is SABRE, and likely they contract for some of SABRE's other services like crew scheduling, but other parts are pure Delta.

Delta uses Worldspan as its GDS, not Sabre.

A post in the ongoing outage thread on FlyerTalk mentions a fire in the datacenter.

http://www.flyertalk.com/forum/27032000-post135.html

Why is it reported as a bug rather than a power outage?? There is a huge difference.

It seems a bug in the infrastructure when they switched generators is what caused the power outage.

Official post from Delta Operations - http://news.delta.com/more-flights-resume-delays-cancels-con...

Part of me, thinks that their systems were attacked, but being into security, I've though a lot about how to attack a large computer system like this. Since server failiures are so common, the system should be able to handle N failiures before anything bad happens. If this was an attack, I guess the attacker must have found a flaw that allowed them to instantly compromise any server. But it actually seems more likely that this was caused, merely, by incompetence.

If the attack was possible, than it was just as likely, if not more, that Delta managed to merely shoot itself in the foot.

I tend to think it might have been a software/hardware update failure of some kind.

"Never attribute to attacks that which is adequately explained by bugs."

It's like how some of my relatives immediately suspect a "virus" whenever they suffer any sort of computer failure. Nope, no malware, just typical buggy software.

I'm one of the lucky people who have to wait. The airport personnel tells us it's the system that prints out weather information that's out. Not sure why one would want to attack that.

Considering check-in and reservation status in general appears to be down I'm guessing this outage is relatively widespread.

Assuming what you've heard is correct this seems to span multiple Delta systems.

If it is power issues and some bit of infrastructure deep in the back end took a nosedive, wouldn't be surprised if it effects multiple front end systems.

Considering it's an airline, it's likely either a IBM i or a IBM Z System where they track everything relating to their flights (schedules, reservations, checked luggage, etc). A lot of companies (mine included) seem to thing these devices are high-availability giants and never test what happens should they go down. I know if ours ever took a dump we would be in some big trouble (though we could function for at least 24 hours).

I am having a strong Deja Vu right now. I remember EXACTLY the same scenario happening a ~year ago. Hmmm

http://qz.com/417617/every-united-airlines-flight-was-ground...

and again http://www.pcworld.com/article/2945552/united-airlines-fligh...

and again http://newyork.cbslocal.com/2016/05/29/computer-system-shuts...

> Not sure why one would want to attack that.

Well, that would make it less obvious.

1. Take a short-position on Delta on the stock exchange

2. Sabotage a piece of infrastructure that's important to operations

3. ???

4. Profit!

Attack the power systems which security overlooks. Sounds like this case.

Strangely, their stock price was not affected by the outage and this news at all. Honestly, I don't get it.

https://www.google.com/finance?q=NYSE%3ADAL&ei=ncSpV_m8MsG9i...

Tough day for them. Moving to 'cloud' hosting... it doesn't just make it someone else's problem. It encourages good architecture.

I just don't understand how a large carrier such as Delta can get so much wrong. What's the DR plan? Are they using ASNs? A company is product, people, and computers. This is not the 20th century, flight plans aren't filed by telephone calls for these guys.

I've tested our DR plans and we were operational after 4.5hours, optimally. And we were 3 people.

This outage reeks of PHBs doing lots of kicking the can and not a whole of implementing.

Sounds like your DR plan or your network don't require power. Care to share?

Yes, 2 datacentres, Microsoft's DFS, cloud backups, and staff trained to know what to do.

For the datacentre I have a friend that we worked out a mutually beneficial arrangement. I keep a few spare servers in his racks and he has a few in ours. Just enough to run the essentials.

Testing was literally walking up to the rack and switching off the power. Then documenting what happens next. If you've configured the ASN just right the domain should roll right over to the backup servers. Offsite DR servers needed a little kick to bring up to speed (I can't remember why this wasn't fully automated).

Most of the time was waiting for backup files to restore and verify. Since we were using someone else's network we were shaped down to a small percentage of their bandwidth.

After that we had a debriefing to see what worked, what didn't work. The usual IT stuff.

I was expecting a "computer bug" story on the front page of HN to actually have some interesting content.

It sounds like some kind of single point of failure in the Delta architecture was uncovered.

Southwest had an outage two weeks ago. Its starting to feel fishy.

Would you mind elaborating on what is starting to feel fishy to you and how whatever that is is starting to feel fishy?

Time for Delta to move to AWS

AWS has outages too. There might be some perceived attraction to transferring the operational risk to AWS, but Amazon's agreements limit their liability.

should be on AWS... all the cool kids are doing it

Merged airlines with multiple legacy systems from 1980s. The legacy systems are well battle tested over the decades, but brittle to add new capabilities like mobile.

Only about a third of of from-scratch large corporate and government software systems succeed. Having to hire the cheapest contractor doesnt always work.

finance industry as well... They're sunsetting legacy systems from many years of M&A, refactoring apps, adding API tiers, using novel tech stacks, building in true-HA and cross-region active-active.

Interestingly, 10 flights were canceled on Allegiant today. Including one I'm booked on.

But apparently it is a normal mode of operations for Allegiant. Absolutely horrid airline. https://www.allegiantair.com/travel-alerts

Allegiant a budget airline that's okay when it works and a pile of junk when it doesn't. They fly routes a few days per week out of my city. In a recent cancellation (by choice), they told customers that they could rebook them on the same day next week. Somehow they managed to do this without offering refunds or alternative flights back that day, the next, etc.

I've only flown with them once, but it was a popular route to Vegas. It was fine for the price, especially since they don't surge as much as normal domestic airlines when you book with less than 2 weeks notice.

Reply on news.ycombinator.com