Hacker News Reader

Comment by jakobegger | Hacker News Reader

> Automation overriding pilot inputs during sensor failure is a recipe for disaster.

In 2009 an Airbus A330-200 crashed into the Atlantic ocean killing more than 200 people because a confused pilot kept pulling the plane up, despite loud stall warnings from automatic safety systems [1]. In the discussions of that event, one commenter asked, "Why did [the plane] not override the pilots' inputs and force a pitch-down?" [2]

Designing safety critical systems is difficult. If the automated system fails, the pilot needs to be able to take over manual control. But at the same time, if the pilot does something stupid, the automatic system should prevent that.

I'm not sure how to square that.

[1]: https://www.telegraph.co.uk/technology/9231855/Air-France-Fl...

[2]: https://news.ycombinator.com/item?id=8452080

View on HN

One of the confounding elements of that particular crash was the fact that when the angle of attack increased above a certain (crazy) amount, the stall warnings went away, due to the inability of the system to cope with an angle of attack that high. This confused the pilots into thinking they were going "crazy fast", when actually they were going "crazy slow".

Yes, I point out AF447 as a perfect and horrific example of a problem which is really HCI (human computer interaction) / UX.

The lack of stall warning meant two different things: "not stalling", and "stalling so extremely it can't be sure, so don't report it"; as you say. I wondered if there should be a different sounding alarm for "can't determine". Or, the pitch of stall alarm could rise, such that pilots could tell "which side of the stall" the plane is on, so that entering stall from one side would side start with a very high pitch or very low pitch. (I'd guess there's a better way..)

So, in this case, it's more subtlety more tragic than you describe, because the pilot did correct the AoA, and nose down, but when he did that, the stall warning resumed (albeit from the extreme other "side" of the interval), confusing him: no alarm, alarm resumes, so he thought he "re-stalled" it and he pulled up again. This kept him mentally "stuck" in the extreme stall. They needed to "pass right through" the stall warning window, and come out the other side. They had sufficient altitude for this to succeed.

This is equivalent to a confused deputy variant of a Byzantine failure.

Since you already have allegedly redundant pilots and alarm systems and hopefully sensors, the remaining part is figuring out which failure detector has ultimately failed to work and why. Also if redundant pilots are failing, it's a systematic failure in training, fatal to airlines and whoever designs training simulators.

The most potentially fatal to Boeing scenario is a systematic failure or negligence in terms of redundancy. Hampering pilot training is bad enough. Another potential failure point is a software issue which means negligence in safety critical software engineering.

I disagree. Flying a plane is pretty much a simple state machine with very well defined states. We have hundred thousands of flight hours available and we can also build a very realistic simulator which is used extensively to train pilots. It is should not be a challenging task to build a reliable auto-pilot and in fact we have already one that covers most of the aspects of flying. On the top of this, most of the flight related incidents are human errors.

It doesn't sound that plausible that we can account for all the various things that can happen to a plane in something as complex as the earths atmosphere with a "simple state machine" . Weather can do really wacky things.

You are saying that it is impossible to implement auto-pilot? How does human pilot fly something planes? You think that human mind can deal with infinite states?

Yes the human mind can deal with an effectively infinite number of states (and let's not get into a pointless pedantic discussion about the true meaning of "infinite"). Until we figure out how to build a true AGI then no autopilot will be able to do that.

No it can't. Flying an airplane on Earth is a very limited exercise in every possible dimension and it can be easily automated without AGIs.

You're simply wrong and have no idea how flight control actually works. In particular failure modes aren't simple at all; real machines can break in a huge number of ways and no one could possibly predict them all. Recommend you take some actual graduate level aeronautical engineering courses and then reconsider the issue.

I am not arguing about how machines can fail, I am arguing for that computers can do things that humans can. Usually with higher precision and much more reliable.

Machines can do some things with higher precision and reliability than humans. Coping with unexpected aircraft failures aren't currently among those things.

noselasd ain't wrong. Autopilots can and do disconnect when the weather conditions exceed their ability. Happens all the time, though probably not often in large airliners because the big carriers have people dedicated to watching the weather and keeping their pilots out of trouble. Your state machine will have to do that, too.

You have been misled concerning the accuracy of simulators. "All models are wrong, some are useful" is a helpful quote to remember here. Certified sims are typically very accurate when it comes to avionics and systems. They are often much less representative where vehicle dynamics are concerned.

Even the most heavily analyzed aircraft can produce surprises once they actually take to the air. Look up "Super Hornet uncommanded wing drop". Or look at dynamic scaling tests (https://www.nasa.gov/pdf/483000main_ModelingFlight.pdf) which uncovered two spin modes in the F-15 that were previously entirely uncharacterized. (Not sure if the linked doc covers that instance specifically -- I learned this from speaking with the study PI. And no, most aircraft programs don't get to do dynamic scaling tests.)

All this is to say that a variety of potential non-linear behavior is lurking inside any given airframe, and all the wind tunnels and CFD in the world will merely allow you to model most of the useful operating conditions, but certainly not all of them.

Simulators, just like autopilots, are useful for finite input ranges. If you leave those input ranges, your simulator will precisely and repeatably model a fantasy, and your autopilot will disconnect.

Source: Short career as an aerospace engineer.

And how do you train pilots for these unexpected non-linear events? What stops you to train auto-pilot the same way?

The point of my post was to say that these unexpected events will not come up in your simulated environment, so you simply will not have the data needed to train the autopilot.

The human, by and large, figures it out. We generally accept that risk. But how will the autopilot react to the unexpected event for which you haven't trained it? Who knows?

Reply on news.ycombinator.com