Hacker News Reader

Comment by sneak | Hacker News Reader

Someone is responsible for the organizational units and their processes.

By that logic, almost no individual failure is a firing offense. Many people in the organization are responsible for ensuring that the processes themselves overlap in ways that exclude the possibility of failures like this.

I'm talking about the meta-failure in management's oversight of the processes. There is belt person, and there is suspenders person, and there is a third person who makes sure there are both belt person and suspenders person on staff and that they don't go on vacation at the same time.

Somewhere above the team lead and below the CEO there is someone who didn't do their job.

View on HN

> By that logic, almost no individual failure is a firing offence

I'd probably agree with that. Firing should be for extreme negligence, or sustained underperformance.

>extreme negligence

If this doesn't qualify, what does?

I would say this is negligence, but perhaps not extremely so.

You are all very quick to judge. I don't work for Google so I don't know what happened, but consider the possibility that this may have been a freak accident. That piece of code for example might only be on a path that is not activated when testing, or it may be executed but not have the same fatal effect.

And before anyone comes along and says "well, every possible path should be tested then": That would be absolutely impossible even if the halting problem wasn't a thing (which it very much is). Even if you somehow have utopian "100% code coverage", that does not mean you have tested every possible input resulting in every possible state combination.

I bet every one of us has a story about an old bug that only manifested years[1] later because of the confluence of many unfortunate things, one that takes half an hour to tell.

[1] Or even decades, in some cases that then made it to the HN front page.

This is the sort of thing that most linting tools will catch, so it's a bit worse than that I think. You shouldn't need any test coverage to catch this.

There are freak accidents and then there are complete lapses is basic critical functionality that demonstrate a total lack of competence.

The article indicates that every person who updated to the latest version could not login. This is a bug that clearly presents itself, every time, in essentially every use case, and is perfectly reproducible. It affects critical functionality that any layperson can recognize is one of the most critical components of a general purpose computing system. The greatest degree of scrutiny should be applied to this system, yet what we see is the functional equivalent of a car factory forgetting to put a wheel on every car it produces.

Sure, there could be a freak accident that makes the 4th wheel robot malfunction, but other stages of the process should catch such an egregious visible failure. Any process that would allow such basic errors to occur without checking or fixing them elsewhere is terrible on its face.

If that is too abstract, imagine you hired a construction company to build a house and they forgot to put in an entire wall. Maybe you just got unlucky and you were the 1 in 1 billion person who gets such a stupid error to happen to them, but the much more likely theory is that they are grossly incompetent, or at the very least far less reliable than you were led to believe.

I don't know what you are trying to add. Yes, if testing would have uncovered this and the release was put out without that testing, then someone is incompetent. But my point is exactly that the "egregiously visible failure" as you say may not have been visible at all during testing. You seem to think that by "not possible to test all paths" I mean that "logging in" was not tested. That's not at all what I mean.

I mean that even for basic user flows, there is an astronomically high number of possible states and external inputs such that, taken together, something that worked fine during testing may stop working at some point thereafter, or in the hands of the actual users. Latent bugs are called latent for a reason.

One point that doesn't seem to have been explained yet - mentioned in the OP article - is that Chrome OS has canary, dev, and beta channels prior to stable, with real users.

For something like this to have happened without those non-stable channel users experiencing it, it seems like there must have been a code change that was pushed direct to the stable channel - which negates the purpose of the other channels in the first place.

At the very least it seems like a severe process failure. I don't think the dev who changed the code is necessarily at fault - it's more a question of how it was allowed to get into the stable channel without going through the normal process.

This isn’t extreme negligence of any single individual. If there are linters/compiler warnings that should have caught this, this is an organizational failure that can be remedied and the manager/TL should be taking a bit of a hit here as having those enabled is best practices. Since it’s ChromeOS I imagine there’s a dedicated team managing the build infra and it’s not using the base set of things that Google3 protects you with, so the fault may lie in that organization a bit as an oversight.

Then there’s obviously the automated testing strategy, manual testing strategy and sanitizer strategy that missed this.

The net result is extreme negligence but to me this kind of failure speaks to several layers going wrong at once rather than one person fucking up badly. Organizationally you try to defend against these kinds of failures with multiple layers of defenses.

Now if this was a malicious internal attack that should be investigated but I’d hate to have to seriously reprimand anyone who typo’ed & instead of &&.

The GGGP is talking about failures at a higher level. Not reprimanding the coder who missed an ampersand, but the manager who was ultimately responsibly for ensuring there was a functioning process for catching this sort of nonsense.

Reprimanding is one thing and firing another though. I certainly think the manager in charge deserves a reprimand.

I wouldn't even say a reprimand is necessary here.

Programming safety is like safety in auto racing or rock climbing: you build the safest thing you can, but the activity you're doing is inherently unsafe, and it will generate failures despite the presence safety measures, so you do your best to study those failures to mitigate them the best you can.

The only time adverse action like a reprimand is necessary is if the manager or tech lead don't take any action to prevent future issues. The only time a firing is necessary is if the security issue was created with intent, rather than accidentally.

This specific type of failure is evidence that they didn't build the safest thing they can.

Linting catches this.

In situations like these there's not really one manager, so who are you reprimanding? I'm sure there's failures all around. Seriously, blameless post mortems are a cultural thing you need to adopt. Reprimanding just instills fear which causes people to take actions that you don't want organizationally (shifting blame/responsibility, hiding issues, politicking etc). Better for people to just be proactive about when a problem exists & marshal your org to prevent such issues in the future. It's unlikely to be the cause of 1 individuals incompetence so any individual(s) you find to blame are just victims of circumstance.

Where you do want to reprimand is if the technical team was repeatedly raising concerns related to failures like this & their concerns weren't being addressed OR someone was being deceitful & hiding issues. That's about the only cases where you want to fire/replace anyone in my book.

The way you fix organizational/process failures is not by firing the person in charge of them every time they fail to catch something.

How many breaking bugs have _NOT_ shipped due to the current processes. Your approach suggests that there is some perfect process manager out there who will let no bugs like this ship (whilst not dropping any other business priorities mind you) that they just haven't hired yet into this position.

This isn't as big of a bug as people on HN want it to be. A few months from few people will remember, and even fewer will care.

> By that logic, almost no individual failure is a firing offense.

That's pretty close to how it should be. That's the core premise of the no-blame post-mortem culture that's great about some companies.

As long as you go through the port-mortem process, which includes doing your best to prevent this from happening again (which may include changing organizational processes, testing strategies, etc), then everyone can move on with their lives.

Firing happens when you have a history of performing below your level. Never for a single engineering failure.

This isn't an engineering failure. Everyone writes bugs.

Engineering process is still engineering in my book, but if it helps:

Firing happens when you have a history of performing below your level. Never for a single process failure.

There is a story (perhaps apocryphal) of a salesman who lost a $10 million deal for IBM. He was sure he was going to be fired. Some bigwig (Watson himself, maybe?) talked him through what had gone wrong. The salesman asked, "Aren't you going to fire me?" The bigwig said, "I just spent 10 million dollars training you."

> By that logic, almost no individual failure is a firing offense.

Correct.

Reply on news.ycombinator.com