Triple-Triple Redundant 777 Primary Flight Computer (1996) [pdf] | Hacker News Reader

Triple-Triple Redundant 777 Primary Flight Computer (1996) [pdf]

citemaster.net

I worked some of the follow on projects to the 777 that involved the next generation 737. A lot of people may not realize this, but much of the software and avionics that goes into Boeing aircraft is designed, built, and manufactured by other smaller third parties. The amount of work that goes into those embedded systems is truly impressive. Additionally, the rigor with which it is built is exemplary. Software standards for commercial aviation are extremely strict, disallowing the use of dynamic memory allocation, pointer math, and other typical low-level language "features" or "shortcuts". There are also strict code coverage test requirements, demanding that primary partitions of the flight avionics have their code tested in what is known as MCDC [1] Additionally, I/O code that uses low level data buses must be tested for bit independence. The process is extremely rigorous.

[1] https://en.wikipedia.org/wiki/Modified_condition/decision_co...

Can you talk about the roles of people who enforce the standards? Is there just a typical QC role like in the medical device industry, or are there extra roles to verify more specialized standards, in addition to QA personnel?

Sounds very similar to the process required by the team who wrote software for the space shuttle. Very impressive.

This is a great reminder that some things are, in fact, complicated. Furthermore, smart people, good process, and hard work can conquer this.

Lately I've been flying a lot and even now, after hundreds of flights, each time the plane takes off I think about how amazing it is that this whole air-travel system works so reliably and safely. Kudos to Boeing and Airbus engineers.

I've worked in nuclear energy, surgical medical imaging devices, and in the auto industry - and it really gets me in a tizzy when a fresh "agile MBA manager" expects a "quick hack" solution to some of these problems. There just isn't always one when quality/reliability are requirements, not nice-to-haves.

Nothing like a good "engineers are gods and MBAs are dumb" post to start the day.

Sounds like you company is clearly hiring the wrong MBAs, maybe the problem is your company :)

Sounds like you haven't worked in nuclear energy, surgical medical imaging devices or the auto industry.

Agile is designed to get a product to a customer, rapidly and adaptively. But this is mostly useful for software that stands alone; you can't always rapidly adapt microcontrollers, bone drill burrs, heavy turbines and optical sensors. Even if you could make a 'quick hack' to one, there's 50 standards to reassess, and if one fails, you have to go back and re-do your change.

The OP is right. There are problems that require longer term, complicated solutions that have to be designed first, approved second, and implemented third. An agile MBA is not dumb; they just don't all work on the same problems.

To be honest sounds more like you're talking about agile people. Move fast and break things doesn't work in any of those scenarios, I've seen a lot of people during my engineering days who just wouldn't work under the pressures required to design a system with extremely high reliability.

It goes back to hiring the right people, not to the people themselves :)

> I've seen a lot of people during my engineering days who just wouldn't work under the pressures required to design a system with extremely high reliability.

I think the difference is that most engineers understand when high reliability is necessary, even if they can't deal with the pressure of implementation. An MBA whose job is to "move the ship faster" simply isn't going to understand the distinction between parts of the system that can be thrown together, and parts that absolutely can't.

Hard to hire the right people when the majority of MBA's aren't engineering undergrads.

From a quick scan - each system also has a different sort of processor: AMD 29K, Motorola 68040, and Intel 80486

Now that is impressive.

Possibly in order to use three different architectures to avoid any nasty arch-specific bugs

I believe these are requirements, that each of the separate teams working on these projects have to use wholly different suppliers to also minimize the chance that the same external factors negatively influence the system.

Also, different compilers:

Each PFC channel contains three dissimilar processor lanes, and software from Ada source code using three different Ada compilers to provide triple dissimilarity.

It was once explained to me that the development of the flight computer software is also done by 3 completely separate teams. Because of this, the chance that the same bug occurs in all 3 systems is incredibly low.

Faults in human cognition aren't randomly distributed. While n versioning helps with these issues, it doesn't prevent them to the level one would assume. Nor does it correct for systemic flaws in the specification.

That's why I only use software coded by at least 3 different species, two of them being not-earth-based and one not-carbon-based :)

Programmers in the future will spend the majority of their time debugging code written by AIs

In Firefox 4, there was a need to implement the HKDF crypto algorithm in a variety of code bases: Java, Javascript, C using OpenSSL, C using NSS. One team coded three implementations. Later, another team coded an implementation. But, that last implementation didn't interop with the first three. Of course, the two teams disagreed about which was wrong, each believing that they were right and the other team was wrong.

Surely if three different implementations in three different languages agree, then the problem is with the fourth, disagreeing implementation! However, the second team annoyingly pointed out that they had automated tests that covered the official test vectors. Well, maybe the tests were coded wrongly too! Well, could the first team implement the tests that include the official test vectors? What a waste of time! There are automated tests that check that the first three implementations all calculate the same results for many random inputs!

It turns out that the very first implementation, in Javascript, had a bug, and that bug was transliterated into two other implementations. But since the second team hated reading Javascript code, and because that team consisted 100% of annoying (but good-looking) pedants, they implemented the algorithm by reading the spec. instead of by copying the first implementation, and so they got it right.

Incidentally, the bug was that the key and value in an HMAC calculation were swapped. HKDF uses HMAC in a counterintuitive way, where the key isn't (necessarily) a secret, but the value is.

This field is old and interesting

http://www.inf.pucrs.br/~zorzo/cs/n-versionprogramming.pdf

https://en.wikipedia.org/wiki/N-version_programming

This is an excellent overview http://www.adelard.com/papers/divchap.pdf in regards to https://appsrv.cse.cuhk.edu.hk/~lyu/book/sft/

Reply on news.ycombinator.com