Software bug fingered as cause of Aussie A330 plunge

http://en.wikipedia.org/wiki/Error-correcting_code#List_of_e...

"When the data from the first two ADIRUs is inconsistent then the flight computer uses the data it received 1.2 seconds previously. However, the software couldn’t handle the two ADIRUs putting out data spikes that were more than 1.2 seconds apart and this caused the plane to behave erratically."

I assume _this_ is the error referenced in the title. Whatever caused the black box to fail, the result was that the code afterwards failed in software, as far as I read the text.

http://www-physics.lbl.gov/~spieler/radiation_effects/rad_tu... [PDF]

Wiki link:

http://en.wikipedia.org/wiki/Radiation_hardening

One way is simply to use larger transistor, so that errors are more difficult to create in the first place.

These components are indeed redundant; from the sounds of the report, one of the ADIRUs malfunctioned (there are three), sending incorrect angle-of-attack data (as well as a lot of other incorrect data). Although the flight computers have monitoring routines for identifying and disregarding erroneous inputs, there was an unforeseen scenario in which a very specific pattern of erroneous AOA data could avoid detection - and this happened here.

It's important to note that the plane was not in autopilot at the time - the autopilot disengaged itself immediately upon detecting an input discrepancy (several seconds prior to the actual pitch down event that caused the injuries) and the captain took manual control. However, due to the fly-by-wire system, the flight controls are normally processed through a rather complex system that (under normal circumstances) helps prevent accidents due to human error (eg, it's impossible to stall an Airbus A330 under normal conditions; the plane will respond to your commands until just before it stalls, at which point it'll force a pitch down or thrust increase to maintain lift).

This invalid AOA data made the flight computers believe the plane was indeed in a dangerously steep climb, in danger of a stall (the stall warnings even sounded multiple times during the flight), and automatic protections called for a pitch down movement (ie, a descent). Due to the precise timing needed for the AOA fault to go unnoticed, some of the flight computers were unaffected - as such, shortly after each of the two pitch down events, flight computer error detection systems triggered failovers - but not until the flight computer currently in charge triggered a descent. The second time this happened, none of the primary computers were eligible for controlling pitch angle (due to each having a pitch-angle-related failure already) and so control passed to a more primitive backup system - incidentally disabling the pitch angle protections that had caused the problems, and allowing the crew to regain control.

A novice was trying to fix a broken Lisp machine by turning the power off and on. Knight, seeing what the student was doing, spoke sternly: "You cannot fix a machine by just power-cycling it with no understanding of what is going wrong."

Knight turned the machine off and on. The machine worked.

In some cases, they say it's not so much a "reboot" as a "boot", i.e. power fluctuations cause the system to shut down. In other cases, it sounds like the power fluctuations cause hardware problems which the software didn't anticipate. And yes, they also suggest there are times when it's just a good old three-finger-salute with no identifiable cause.

What actually happened is one (of three) ADIRUs started mixing up its outputs - it intermittently labelled altitude data as angle-of-attack data. This happened after internal checks for data consistency, and before CRC generation - exactly how this occurred is a mystery, but it happened without the ADIRU detecting an internal fault.

Now, AOA data is measured by multiple independent sensors, and each ADIRU has its own sensor, on different sides of the plane. Because of this, under normal circumstances, AOA data can be somewhat different between the ADIRUs - much more so than the other inputs. Moreover, two sensors are on one side of the plane, and one on the other - making the median method weight one side far more than the other. As such, the flight computer software normally averages the AOA data from all three ADIRUs, but has an error checking system to detect and exclude erroneous data. This system detects when an outlier occurs, after which point it enters a 1.2-second cooldown, in which it uses last-known-good data. 1 second into the cooldown, it rechecks its inputs to see if the outlier remains (and if so kicks it out of the system), and at the 1.2 second mark it resumes normal operation. This error-checking algorithm is supposedly meant to deal with the situation where two ADIRUs are both temporarily bad due to some turbulence on one side of the plane - a fault scenario identified during early testing of the A330.

The key thing is, at that 1.2 second mark, it assumes any input that passed the 1-second-mark test is valid. In this event, ADIRU 1's AOA input spiked, triggering the 1.2-second cooldown. It then returned to a normal value for the 1-second-mark test, and then spiked again - at the 1.2 second mark, this invalid data was sampled, the average of all three ADIRUs taken, and a bogus value resulted. Once the spike resolved again, rate-of-change limiting prevented the value from returning to its true value immediately.

Note that there's also an independent flight computer monitoring system for each flight computer that only listens to one of the three ADIRUs, cross-checks the control outputs, and has the authority to disable the specific flight computer it's attached to if there is a discrepancy for too long (450ms). This flight control computer monitoring system did work properly, and disabled the flight computers in the end, switching over to manual pitch control via a more primitive, secondary flight control system. However, it took two faults (ie, pitch down events) to do so - one to take out each of the primary flight computers with pitch control authority.

In any case, a new error detection algorithm that avoids this issue is currently being distributed as a software update, and in the meantime there are safety bulletins instructing pilots to disable the affected ADIRU manually at the first sign of trouble.