Hacker News Reader

Comment by gnfargbl | Hacker News Reader

> the bug was missed because it only presented when many aircraft at the same time were using the system.

How the heck do you end up with a concurrency bug in a system where the calculations appear to be logically independent?

View on HN

A concurrency bug that got fixed in production 5 hours later. It must be very unusual code or some people are excellent at debugging concurrency problems.

Article says that bug was introduced in an update that morning. Somewhere there was a developer familiar with the code and exactly what changed. Wouldn't be surprised if he smacked himself on the forehead and knew immediately what the issue was when he heard about the bug.

No it doesn't say that.

> That morning, a software bug in an update to the DynamicSource tool caused ...

This says when the bug happened, not when the update was made.

Or they just rolled back an update. I don’t think you can do much more in 5 hours.

Right. Even if the developer knew exactly what he did -- any sane provider would put the updated update through a full QC cycle, which would take a while I'd figure.

This sort of thing - in my uninformed opinion - should not be a service, and should be validated strictly as though it is part of the avionics software. After all it sends data that directly affects the power settings the pilots will use just as if those power settings were directly affected by the output of the program. That makes it an integral part of the plane to me.

I believe the tool takes data from multiple sources including airline reservation systems, baggage handling systems, refuelling systems and cabin crew confirming people are correctly seated on handheld devices and validates it for consistency before performing a standardised set of calcs. In which case the safety argument for having that airgapped from the flight computer, which receives only a few final calculated values (manually input by a pilot who also can potentially spot outliers) is probably stronger than the safety argument for integrating everything or manually inputting a large number of values into the flight computer (creating potential remote attack vectors and increased likelihood of human error respectively).

Would have thought lower than ideal safety margins on takeoff speed (the calculations were apparently <20% off the true value) were a more concerning problem

> the tool takes data from multiple sources

And 90% of those sources are ancient balls of mud that spit out stuff that looks like

    !ORD 06/001 ORD RWY 04L/22R CLSD 2106231700-2106232300

    KAUS 092135Z 26018G25KT 8SM -TSRA BR SCT045CB BKN060 OVC080 30/21 A2992 RMK FQT LTGICCCCG OHD-W MOVG E  RAB25 TSB32 CB ALQDS  SLP132 P0035 T03020210 =

Excuse me sir... that language is simply inappropriate here!

The fact that a lot of this output is almost complete inscrutable means you have to build a tool or something to translate it from that back to something a human can quickly read/grasp... and translation mistakes can happen there too :/

Yes, mistakes can happen there too. And mistakes can happen during data entry into FMS. It's a miracle that all of this works most of the time.

TaaS -> Tailstrikes as a Service...

Make fun of it all you want, but the whole point of that is that it is machine readable and concise.

`KAUS 092135Z 26018G25KT 8SM -TSRA BR SCT045CB BKN060 OVC080 30/21 A2992`

That is a very concise way of conveying a lot of information about the weather. It doesn't take long to learn it, and at a glance I can tell you that the weather is kind of shitty for flying at KAUS, and I can plug it into any flight app and read it in plain text if I really want.

Someone forgot their 'if err != nil { return err }' check from their microservice call (whenever it happens to be overloaded) and just plowed through with some intermediate weight component being zero?

"On error resume next" is a perfectly fine default error handling strategy.

Especially for mission critical software that lives depend on.

It could be something as pedestrian as the software writing and later reading back from /tmp/somerandomfile. Or retrying with similar, predictable names.

Resource contention causing part of sum to fail to be incorporated.

That seems plausible. E.g., if the package cargo API timed out it could be treated as a 0.

It does seem plausible, but also almost heinous design if true.

Yeah the more I think about it the more this makes sense to me.

Both were lighter than expected (so if it was a swap with another aircraft we wouldn't necessarily know - possible that 4 aircraft got the wrong data and we don't know the other two).

But it feels more likely that the sums just failed and they incorrectly caught the failures instead of letting it fall out. catch(e){} is a bug factory for sure.

That was my guess as I've seen people use non fatal timeout conditions and keep processing when a value isn't returned from an external data source.

Maybe it sent the result of a different aircraft? So each calculation was still correct, but they got mixed up when responding.

I've thought about this - if this is the case, they are extremely lucky because different planes could have significantly different payloads. Of course, then the TLAR would signal an issue.

Also this would mean some other aircraft had a particularly powerful departure, since both aircraft had lower-than-true weight calcs.

It's almost like we should try to stick to languages where creating concurrency bugs is nearly impossible, thereby pre-empting an entire class of bug that might have resulted in deaths

What language would be a good example of that?

any functional one with immutable data would suffice. my current favorite is Elixir, but others fit this bill.

There is still the matter of dealing with mutable stores like databases, but well-written code would make that impossible as well, plus permit tests to run concurrently

Reply on news.ycombinator.com