Comment by munificent | Hacker News Reader

Comment by munificent | Hacker News Reader

There's a weird vibe in this article I don't like. It (correctly) notes that as the number of anomalies that the automation passes on to the human operators goes down, the rate that humans successfully handle them also goes down.

But it doesn't seem to do a good job of clarifying that the total number of incorrectly handled anomalies is still decreasing. Let's say your automation goes from handling 90% to 99% of the anomalies and that when it does handle one, it does so correctly. We'll say that the increased rarity of human interaction and the inattention and weakened training that causes makes the human pilots to go from being able to handle 90% of them correctly to only a terrifying 40%.

Let's run the simulation. With the old automation:

    1000 anomalies occur
     100 (10%) make it past the automation
      10 (10%) make it past the human operators

So 10 catastrophes. Now with the new moderately better automation and much worse human performance:

    1000 anomalies occur
      10 (1%) make it past the automation
       6 (60%) make it past the human operators

6 catastrophes. Even though the human performance was much worse, because they are the last stage in the pipeline, it has a lower effect.

Now, I just pulled these numbers out of my ass, but I think it's important to focus on the total number of automation+human failures and not single out one stage or the other. From the passenger's perspective, they don't care who saved their ass, just that it got saved. If we can make one stage more failure proof at the expense of the other, it can still be a net win.

Also, there are things organizations can do to combat the problem of human operators getting rusty. Practice. A lot of organizations just don't do it though because it's too tempting to view the automation's cost savings as "free" and just take them for granted, but it can help a lot.

People get rusty for a reason. Your solution suffers from a couple of problems:

- Practice isn't the same thing as actual events. Being good at practice is more likely to diverge from being good at crisis response as actual crises become rare, because the criterion of matching what would happen in a crisis gets much less important. Thus, peacetime militaries often need radical overhauling before they can really get much accomplished when war breaks out. (Also consider - we have a lot of people who are really interested in how medieval combat (whether in a battle line or a duel) worked, what effective use of weapons looked like, and so on. But for all the discussion, we don't know, and we can't know unless we actually stage regular battles-to-the-death with period tooling. One form of combat practice, however, has been preserved as European fencing. How closely does it correspond? Again, we don't know, but consensus is "not well".)

- The automation's cost savings are free -- in fact, in this example, they have a large negative cost, cutting catastrophes by 40%. Keeping everyone in shape to handle crises they're likely to never actually see is, arguably, an enormous waste of money. (In addition to actually being impossible much of the time, as in my first bullet point.)

on empty legs (flights back with no passengers) you could disable automation as protocol.

A very insightful idea. You could also measure this by ensuring pilots have a certain number of hours logged without automation (practice by pudding).

Are there analogies in other industries to this approach?

Never pulled the plug on your UPS to make sure you can or ripped out half the raid array without warning?

> Are there analogies in other industries to this approach?

https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

- Yeah, practice is definitely not the same, but depending on the domain, it's usually at least a lot better than nothing, even if the real thing is a lot better than practice. (To speak to your example, at least in wartime, peacetime militaries don't have to worry about reteaching their soldiers EVERYTHING. Maybe they've never been in real combat, but at least they can consistently shoot at a target. That's better than not being able to do that either.)

- I mean, it depends on the specific case, right? Obviously that's true in this example. You can also come up with opposing examples. I also misspoke a bit -- even if the automation cost savings are free and you're strictly better off with it than without it, adding some practice for human operators may get you even more savings, and it's often overlooked.

My point is just that practice/drilling can be a useful tool in the toolbox. It depends on the situation, but it shouldn't be ignored.

First responders practiced for handling scenarios like the Boston bombings for years before 2013. It paid off.

Initial responses to ebola in the US tended to be pretty badly bungled because essentially nobody was trained for handling very dangerous, highly contagious disease. Should they have been? Going back how long? Should they be now?

> To speak to your example, at least in wartime, peacetime militaries don't have to worry about reteaching their soldiers EVERYTHING. Maybe they've never been in real combat, but at least they can consistently shoot at a target.

My understanding, which is only very loosely informed, is that while soldiers get retrained, peacetime generals, who make the decisions, usually need to be replaced. Crisis response, depending on the crisis, will vary in how much it demands one or the other skillset -- but I think someone whose job it is to oversee safety systems, when in practice (1) the safety systems almost never fail, so that (2) the job consists mainly of convincing politicians that they should feel good about what you're doing, can be reasonably closely analogized to a peacetime general. Making a contingency plan that sounds good to an audience with no experience is a different skill than making a contingency plan that effectively addresses the problem.

Practice definitely can be a useful tool. But just as there are situations where it will help, there are plenty of situations where it won't. When events are all drill and no reality, mission drift in the drill is inevitable. Sometimes a pound of prevention is worth an ounce of cure.

"Keeping everyone in shape to handle crises they're likely to never actually see is, arguably, an enormous waste of money."

This point, while true, depends heavily on the consequences of the crisis. If the cost of failure is too high, it's not a waste of money.

Practice isn't the same thing as actual events. Being good at practice is more likely to diverge from being good at crisis response as actual crises become rare

Had the crew of Asiana Flight 214 been properly trained in manual landing, lives could have been saved. Practice isn't the same as actual events, but it does produce measurable and valuable results.

You can verify this for yourself with something as mundane as giving a presentation.

Even though the human performance was much worse, because they are the last stage in the pipeline, it has a lower effect.

Minor mathematical nit: it's not because of which stage of the pipeline you are in. Multiplication is commutative.

Reply on news.ycombinator.com