Hacker News Reader

I did a Five Whys for Amazon once. It had been an embarrassing incident for the team. We had put a huge new feature together over a few months, and when we turned it on...nothing happened. The whole feature had been entirely disabled by a typo I made in a configuration change. At the time, we had a test team that was tremendously overworked and tested things almost entirely manually. They asked for a couple extra weeks to test our new feature but eventually greenlit it. My Five Whys basically asked "how did this get to production if it didn't do anything?"

My manager handed it back to me and said "this takes the blame off of you for this mistake and puts it on the testing team. Ask different questions."

So I wrote another one that was 100% focused on me. "Why did this happen? Because the config was wrong." "Why was the config wrong? Because I made a typo." "Why didn't I notice the typo? Because I didn't run the feature after making the change." "Why didn't I try it out? Because the feature is very difficult to invoke or observe." "Why is the feature hard to invoke directly?" "Because <x,y,z>."

What I learned from that is that the problem with Five Whys is that there are many different branches of questions that are worth asking, but that design only allows for exactly one path. Choosing which one is an editorial decision on the part of the author. The outcome of a 5-Whys document is, by its nature, one change. But most of the time, outages at mature tech companies are the result of several different problems coincidentally lining up (if exactly one thing going wrong caused a major problem, that itself is the second problem), and a postmortem process really needs to examine each of the things that went wrong separately.

I has a similar experience where I had to 5 why a issue and... the gist of it was that the why was because I bungled it up. I just.. saw it as an issue while I wrote my stuff, added a TODO and then... just never did it. So then, after the first 2 why's I could only answer in the back of my head, "Because I'm fucking stupid, that's why!"

I get the idea with the why's but sometimes, it's just that simple, you can't really go deeper than that. I fucked up.

I think you can still usefully push past the "I just never did it."

If the problem is that you are overcommitted and things are starting to slip, that's a different problem than "I got distracted by something shiny and forgot." In the first case, it might make sense to try to grow the team, for instance.

The question then becomes: what could you have done for the "system" to automatically prevent you from forgetting that? Would an end-to-end or smoketest prevented that? Would having a TODO-linter have stopped this? Would singing your country's anthem from the top of a hill prevented this (I dunno, maybe it's a really magic song)?

You are looking for improvements to your process that will help you protect yourself from yourself. :)

Exactly this. We have a saying in the GxP QA world - look at the process, not the person.

Anyone can make mistakes, and sometimes the naive 'why' of something is just that someone made a mistake. But if you focus your questions on the process, it becomes 'why did the process allow this mistake through'. Perhaps a checklist could have helped. Perhaps you need a QC [1] step in the process. Maybe the instructions aren't detailed enough.

[1] Though there are many sometimes subtle differences in how these terms are used across industries, generally the difference between Quality Control and Quality Assurance is that QC is part of the process (e.g. a built-in check in the system), while QA is a separate process making sure these checks and balances are working properly.

To bring in the car analogy, QC would be your tire pressure sensors, and QA would be your mechanic checking that the pressure sensors are working correctly.

You can always go deeper.

Why did you fuck up? Because you didn't get enough sleep. Why didn't you get enough sleep? Because you were on-call at 3am. Why were you on-call at 3am? Because the apps keep crashing once a week. Why are the apps crashing once a week? Because nobody has prioritized fixing them. Why has nobody prioritized fixing them? Because the feature work is higher priority.

Or: Why did you fuck up? There was no validation of the change before applying it. Why was there no validation of the change? Because we've never prioritized validation. Why have we never prioritized validation? Because we don't think it's important. Why don't we think it's important? Because we never look back at our previous failures to see if there's a pattern of failures from lack of validation.

So even from "I fucked up" we find a number of problems and things we can start addressing.

> "Because feature work is higher priority"

Major cause of burnout and incidents, yet management will ignore this particular "why", because their promotions are based on features, mostly. Also why it sucks being an SRE unless the company empowers your team.

The hardest part is usually figuring out a reliable way to avoid making the same mistake in the future. "Don't make mistakes" is never a viable solution to that problem.

Yeah, I don't think blaming the typo is the right outcome for this. The right outcome would be identifying where your testing / review / launch / validation process failed. Evidently it failed, or nothing would have gone wrong even when you made a typo.

Usually the response to a good root-cause investigation isn't "I'll try to avoid making mistakes" (although sometimes it's just a mistake and no changes is required). The outcome should be a new rule or practice for the organization, some kind of decision that determines how things will be done going forward to make the error less likely. This almost always requires a leader at some level declaring by fiat that things will be done this way going forward; if your managers aren't doing that, they're not 'leading' at all.

I don't think that's an issue per-say. You can't fix everything, but you can at least try to find the most likely path and do something about it. Even if it wasn't the biggest cause, you made progress towards improving some of the variables at play for the issue.

There's clearly something to learn and improve in your 5 why's. There was a hard to test feature, typos were allowed to pass through validation in the config, there was only one person making the config change and seemingly no one to review the change, etc.

Ok so all these seem like they'd contribute at least somewhat to why this issue occurred. Sure there might have been other deficiencies, like why didn't the test teams also forgot to test the feature or thought they did but didn't, etc. But you can't address everything.

So maybe you update the validation code for the config so it is more likely to catch such typo in the future. Maybe you make improvements to your config change management and enforce a 2-eye rule on config changes, and a 2 person review on the change. Maybe you make improvements to your testing capabilities so that such hard to trigger features become easier to write an automated test for. Maybe the leadership learns to inssist on planning more time for dev QA and writing tests when launching the next big feature, etc.

Toyota's implementation of Five Why's actually doesn't stick to just one path, they can branch out or do multiple of them. But you're right about multiple causes and the need for alternative strategies to find and fix causes.

The problem with process is that most people follow it as prescribed rather than following the spirit of it.

This is not accurate. Im currently working on a "5 Whys". I am writing it in a nested bullet format which allows multiple branches. Example:

  * Main Why?
   * Why 1?
    * Sub Why 1?
     * More Whys
   * Why 2?
    * Sub Why 2?

Its totally fine to have more than one change or path. Theres nothing in the nature of the doc that requires one change or one path, thats just an assumption.

The "theory" of 5-whys is that with a proper mindset and the right questions (coming out of that mindset), you can find a root cause to anything in at most 5 linear steps.

You are certainly free to adapt it in whatever way you wish, but you don't call that "5 whys" anymore. TBH, your approach looks closer to fishbone diagrams instead.

>The "theory" of 5-whys is that with a proper mindset and the right questions (coming out of that mindset), you can find a root cause to anything in at most 5 linear steps.

I have never heard of this before. I had heard that the 5 why's was a way to force a person or team to be more than surface level about why a feature is needed or a problem happened.

And also, ONE way to represent the work was through a fishbone.

But part of the 5 why's is that multiple "why's" converge (e.g. why did this break? because we have no change control... why does X process suck? Also, because we have no change control)

My favorite feature of 5 why's is they push you into going deeper. If a process keeps failing, the 1st level reason for the failures can be different (X person changed the data, dates suck, the primary server crashed), but the 3rd or 4th reasons should show some similarities and ideas on how to prioritize (e.g. if we have automated unit tests, it would have prevented 25% of our outages!)

This is wholly incorrect. The internal tool in which you write these 5 why's specifically says:

"5 Whys" is a helpful mnemonic but authors are encouraged to use more than five causes as they complete their root cause analysis

A better way of doing this would be the Ishikawa diagram.

https://en.wikipedia.org/wiki/Ishikawa_diagram

It isn't better, it's different.

There are a plethora of tools that go into a problem analysis, six sigma, lean, or whatever labelled toolbox.

The important thing is being able to use the tool, not just do it, as the parent mentions in their comment:

It's easy for a manager to say "do a 5 whys". It's another to be able to support fixing what it finds out. Or if not your manager and just you or a small team with seemingly everything in your control to introspect and challenge assumptions or prior work that might need rethinking. That's the hard part.

> "Why didn't I try it out? Because the feature is very difficult to invoke or observe." "Why is the feature hard to invoke directly?" "Because <x,y,z>."

Now that's engineering.

Code or any feature that's hard to get in isolation and test is prone to fail in unexpected ways. And if the failure in prod is big enough that gives you the opportunity to show that it's cheaper to invest engineering time to make this part of the product better instead of picking up the pieces later on.

I worked at Amazon too. This outcome of a 5 why,

"Why?" "Because our culture has product managers demanding things with short deadlines and no concern about the code quality."

Does not sound like anything would change. I could chalk up a lot of failures, missed goals or mistakes to this culture of shipping the bare minimum because that's all the time you have. Sometimes it works well, others lead to tech debt that metastasizes because the engineers don't have to use what they built leading to instances like pricing errors that lost significant amounts of money.

Working at Amazon sometimes felt like everyone was gaming the system to hit impossible numbers/deadlines - Goodhart's law at scale.

Hah. I cannot count the number of times we made action items as a result of five COEs and the ultimate result was a SIM issue permanently sitting in our queue because there was always something more important to do.

That makes sense. I remember some principal saying "COEs are a chance for engineers to drive work priority" at a coe review.

It worked pretty well in my org (fulfillment/supply-chain) but maybe I just got lucky. As you might imagine that org was pretty functional, being the backbone of the retail business.

In my experience at Amazon, orgs which deal with physical things operate differently from orgs which deal solely with digital things. So the advertising org was more very "move fast and break things", while the Kindle org was more "if we ship a broken build then we brick millions of pieces of hardware, so we're going to manual verification on daily builds". Fulfillment having a healthy/diligent development culture fits that experience pretty well.

> "Why?" "Because our culture has product managers demanding things with short deadlines and no concern about the code quality."

I find answers like this to be the real downfall of a five whys approach. Is it really true that product managers have "no concern about the code quality"? At best, that seems like a statement that the product managers will not accept, but even worse, it isn't directly actionable.

I'd prefer something like, "Because our teams are not given n days for quality assurance before feature delivery", or "because our estimate of 21 days to deliver was replaced with 14 days without our input".

If it isn't measurable, how can you make the needed changes? "Care more" isn't quantifiable.

Indeed, the five Why's is just one of the starting points for introspection. Each person/team/department/org needs to find ways to generate consensus and actionable items out of that exercise. Reactionary phrases like "because x sucks" aren't very useful.

Oh good, we have arrived at the no true scotsman phase. It's not that scrum and agile aren't working, your team just hasn't introspected enough!

If we knew of a perfect system without flaws, we'd already be using it.

Talk to industrial engineers and they’ll tell you these exercises are just a starting point to try and understand the problem. We all like to pretend there are always simple solutions (if I had a nickel for every time somebody told me the root problem was ‘we just need more people’…) but I’m practice, process improvement is often messy and convoluted until you drill deep into the system

At that point, it's not actionable either, it's just blame shifting.

I disagree so strongly with this. There's a pervasive fallacy in the industry that things that aren't quantifiable, or are hard to quantify, can't be real or true. It's just wrong: they can be very true, and often such things are deeply true but whole organizations are ignoring them because they haven't found a convenient way to shoehorn them into their KPIs / OKR framework / whatever.

Yes, it is often the case that product managers have no concern about the code quality. No, the problem is not that N days aren't given for QA. If the product managers talk to the engineers, they will get a sense of the problem. If they work as colleagues they will be in this sort of discussion all the time. When a product is implemented, they'll keep talking, and they'll be aware that the major corners are being cut, and they'll understand that that puts it at risk. Then they can make better decisions to avoid this.

If they have that information and still make the wrong decisions, then they have perverse incentivizes or are behaving maliciously to the business.

And if they try to reduce the intuition and professional opinions of their engineering team to some numbers -- estimates, QA lead times, bug counts -- which are pitched over the wall, badly summarizing actual expertise -- then they will fail, badly, over time, because that is a nonsensical way to work.

Good decision-making and leadership, imo, don't need you to have found ways to quantify things, and your time is often better spent doing good work than waiting for things to be put into numbers. Quantification helps, sometimes.

"If it isn't measurable, how can you make the needed changes? " Doesn't that question sound absurd? You make the needed changes! What does measurability have to do with it?

I strongly agree. I wrote something similar here:

https://demodexio.substack.com/p/a-master-craftsman-will-hav...

Strongly agree. Another issue with the measurement cult: Goodhart's Law.

https://en.wikipedia.org/wiki/Goodhart%27s_law

I'll take the opposite view here, mostly because I think it's an interesting issue and don't know where I fall on it.

If you can't measure the difference, how do you know you've helped? Or if you've made it worse?

> "If it isn't measurable, how can you make the needed changes? " Doesn't that question sound absurd?

No, because you're saying something is needed with no observable difference when it's changed.

If you can't in any way measure the difference between action and inaction, isn't that equivalent to saying "my action is indistinguishable from inaction"?

If someone said "I've fixed the bug, no tests changed and you won't see a difference in the reported issues but it's fixed" how confident would you be that they've actually fixed it?

How do you trust their changes are needed?

There's a big difference between something being discernable and something being measurable. For instance -- suppose you join a company and you can tell the culture is messed up from day 1. You go to the CEO and tell them as much. Should they ask: "can you prove it? where's the data?" or should they say: "tell me more, help me understand."

.. Obviously the latter. If you can perceive it, it's a valid belief. Obviously it is _somewhat_ subjective, but even subjective things can be true, and from someone with good judgment, they usually are. You probably don't need to go run a survey or something to prove it -- just make a convincing case, in words.

Every case is like this one. You can make changes and see that you've had an impact without ever getting distracted by finding a way to put it into numbers, quantification, and measurement.

So you make a change to the culture - did it help? What does help mean here? Happier staff, better retention?

> You can make changes and see that you've had an impact

Well the obvious counter here is how do you see you've had an impact when the impact is so hard to measure?

> You probably don't need to go run a survey or something to prove it -- just make a convincing case, in words.

And yet I've seen the huge issue of wordsmiths with nothing to back it up with taking over and ruining teams.

This is all hypothetical, but suppose it makes for happier staff. Do you think you can best discern the happiness of the staff by.. sending out a survey? Or maybe you'd be able to tell by the feeling of the room in the cafeteria, or the tone of your conversations, or the enthusiasm they seem to show at their jobs, or the messages they send you?

It sounds like "Because our teams are not given n days for quality assurance before feature delivery" is just a further Why down. I can see how this becomes a loop if someone isn't trying to get a real answer though.

If you believe a values problem isn't actionable, you're not thinking for the long term.

It's the foundation of strategy.

"we don't have a method to prioritize code quality"

> As someone who used 5-whys when I worked at Amazon, I love it and the criticisms here don't resonate. It's not about finding one path of causation, or finding everything you can point a finger out. It's about being willing to admit the actual causes and then do something about them.

You're right, of course. To do causal analysis at all, you have to do it respectfully and well. That's the first step.

When you have the first step down, there are other analysis methods that are not five whys that does this better. One of my favourites is Leveson's causal analysis based on systems theory (CAST).

These dig wider as well as deeper, and offer more interesting questions to ask along the way rather than just "what caused this?"

imo, the question "Why?" is really supposed to be more like "How could this have happened!??". It really makes a lot more sense that way.

[edit: to respond to the dead comment about what else "Why" could mean, the difference is the emotional valence. It's not asking the _factual_ question of how it happened; it's applying the appropriate outrage and embarrassment to the fact that it happened. "My god, we messed up! How can we ensure it never happens again, with urgency?".]

Sure, but even better is to be more explicit:

- Which business constraint was violated?

- Why is that an important constraint?

- What control mechanisms did we have in place to prevent that from happening? Have they prevented the constraint from being violated before? Why where those mechanisms insufficient this time?

- What control mechanisms didn't we have? Why not?

And so on. You get fan-out of much higher degree that way, but it also, in my experience, is a much more efficient way to search the causal network.

I don't really agree. The model you describe presupposes a functional framework for decision making -- constraints are in place and respected; controls exist, etc. Most of the time the result of the Whys process is, in my experience: "oh, we really need a framework for constraints/controls/etc around this".

I guess it looks very different depending on the business you're in and how codified everything already is. If you're investigating failures in, say, a manufacturing process, everything is pretty rigid already and you're finding problems with the rules you already have. If you're investigating failures in a hodgepodge software engineering organization, the problem is that there is no rigidity in the first place, and those processes need to be created from scratch.

I'm mostly familiar with the latter, and usually the result of whatever you come up with is a place where leadership ought to be directly applied.

(I'd hazard to claim that: if you do a 5-Whys in an organization and the leader of the organization doesn't see something that they can go fix directly -- not by delegating but by leading, themselves -- either you did it wrong, or they're a bad leader.)

While there are other techniques, one thing to think about is how often the analysis/review group is that particular group. In large organizations, if they particular issue being analyzed is with a group that hasn't gotten together before, a more accessible technique like five whys could be easier to use and mediate. If the group regularly gets together to do such analysis, or has a longer range time frame to complete the analysis, more involved techniques could get to deeper work.

>"Why?" "Because our culture has product managers demanding things with short deadlines and no concern about the code quality."

If you want the 5-whys to not be a joke, you certainly shouldn't be arriving at such strong conclusions without an enormous amount of evidence. There's likely many other explanations.

A serious answer to a "why" question requires enough evidence that it could not be explained in any other way.

If we just want to arrive at strong conclusions that fit in with our priors, so be it, but at least be honest that you don't care about the truth.

When the 5 whys technique is used for actual process improvement, data collection is part of the process to measure improvement of refute a claim. The next steps usually test multiple theories (sometimes concurrently).

The Five Whys are a great as a tool for personal introspection, but in business it seems to become an exercise in chaotic story telling, with the final "answer" highly dependent on the initial assumptions.

Let's take this in a different direction:

> "Why did our code break?" "Because we habitually ship things without thoroughly testing them."

.... or "Because Bob didn't write proper tests for his patch, so the bug it introduced went undetected"

It obviously would suck to be Bob here, but the point is there is no single right answer to any of the whys and each choice along the way will lead us down very different narrative paths.

You can't achieve greater certainly simply by chaining multiple uncertain statements together.

Like any tool, it can easily be misused.

The post in question is a good demonstration of how you can misuse the 5-whys, and it gets well deserved criticism for that. It's obviously not getting anywhere 5 levels deep, so it's asking the wrong questions, and honestly, giving very bad answers that don't lead anywhere: 5-whys require a specific mindset first. Thus, the final "root cause" is very suspect and there's one too many "feel" and "think" in there.

To illustrate, the answer to #2 of "Because I am constantly tempted to check my email, rather than work on important projects." is lacking in substance to answer "why are you getting distracted" (it says the same thing in slightly different words). You can go on-and-on in the same vein. "Because my email notifications pop up and I check them out", "Because when I check my email notifications out, I read the email and now I've lost my context and got immersed into the email topic instead"... That's not helpful at all.

As soon as you get to "I get easily distracted by email", that's already a problem worth solving ("root cause"). And it can have multiple solutions: 1. if you are a blocker for something, move that to an email list of people, a "board" or "forum" discussion; 2. if you are not a blocker, stop notifications and set checkpoints for when you'll read email during the day, and open a side-channel for urgencies if it's ever needed.

If that's not the root cause (it is actually expected of you to be responsive even if you are not a blocker to anything), only then is it worth digging deeper to find problems in the team that make this so.

Of course, I am working with the assumption that being extremely responsive to email is really hurting this person's work (they were not hired to be responsive to emails, eg. as a technical architect to support a development team).

> Then we did none of the things suggested. Well, there's no point. Just quit. You're going to be mediocre forever until the management chain buys in. The reason Amazon scales well and can do so many things is that that their management is incentivized to care about things other than shipping/deliverables/this quarter's returns. It's still about the business overall, but Bezos' long-term-thinking has infected the entire company with long-termism and it's very effective.

There's a lot of cargo culting around Amazon, but having a strong engineering culture and technical leadership is almost never what companies choose to emulate.

True engineering orgs are quite rare (and will often ship disproportionately compared to their competitors. Just think of pre-merger Boeing for instance).

> I am quite sure that Google/Microsoft/etc do not have this quality in the same degree (or they would not have made many of their obvious mistakes). Not that Amazon doesn't make mistakes; they just make different kinds than a lot of other companies

I'm interested what are the obvious mistakes you are thinking about.

> The reason Amazon scales well and can do so many things is that that their management is incentivized to care about things other than shipping/deliverables/this quarter's returns. It's still about the business overall, but Bezos' long-term-thinking has infected the entire company with long-termism and it's very effective.

Is comingling inventory among sellers long term thinking?

I actually worked on that project, and yes, it is. It was a good idea whose implications were not fully understood -- but Amazon will eventually get it right and it will stop causing problems, because they have a good 'make mistakes and fix things' loop. (I think this has already started happening but this moves very slowly.)

The main reason for commingling inventory is to be able to ship a product from the closest warehouse if it's the same as something in another warehouse (if you order it on the east coast and the nearest Amazon-owned inventory is on the west coast but an FBA seller has it on the east coast, ideally you could fulfill the order from the east coast).

The problem is, nobody anticipated the degree to which sellers began cheating this system when they figured it out. I don't think Amazon was ready to fully moderate the whole system against exploitation. They are no doubt working on it but it's a huge battle.

To me your message embodies the criticism of the five whys. You don't run a five whys analysis because it's not effective. You run a more open ended root cause analysis with the goal of understanding issues, which is effective.

I believe that what I'm talking about is what "five whys" _actually_ is, and what a lot of the critics think "five whys" is, which they are rightfully criticizing, is just not a thing at all -- it would be trivially useless and ineffective, and obviously is so to anyone who does it.

In particular 'five whys' when run correctly has neither a set number of whys, nor a particular linear structure, nor a fixation on assigning blame, and involves actually taking actions in response to the root causes, not just creating meaningless action items. If those don't apply, you're not doing five whys, you're doing some bizarre aberration of it.

To me that sounds pretty clearly like not the five whys: there aren't five whys, they aren't linear, and you're not asking "why" at every step. The history of the five whys and Toyoda make it clear that it's 5 linear whys. You're performing a root cause analysis, which I suspect is why you find it effective. VS the five whys are not an effective analysis tool. Maybe where we see things differently is you consider the non-five-whys analysis part of the five whys, and I consider it "root cause analysis". For me, an issue with still calling it the "five whys" is watching other folks less familiar with how to do root cause analysis using the low quality and awkward "five whys", without any clear direction that they shouldn't use that strategy because it's not effective.

Yeah, I guess what I'm describing is "the thing we did in the Amazon COE process that we called 5 whys". Whatever that was called, it's definitely better than what you're describing.

>management is incentivized to care about things other than shipping/deliverables/this quarter's returns.

Could someone elaborate on how that's done?

Management has to be rewarded by _their_ management for delivering on abstract improvements that make things less bad without necessarily pegging them to metrics and returns. Their management has to be rewarded by the managers above them, etc, all the way up to.. somebody. Ideally the CEO because the board ain't gonna do it. Bezos was great at this: it was always clear that fixing your shit was the first priority, because the C-suite understood that for a company that large to scale its labor, teams have to not be blocking / causing friction for each other. Hence being very forward-thinking about abstracting everything out into service-oriented architecture. (although it took like two decades to actually get there completely. My org, fulfillment, was one of the last to do it, and only finished after I left.)

Why did you leave Amazon?

I was 25 and it felt like a weird thing to be doing in my 20s when other people were traveling the world.

Couldn't you travel while still being employed by Amazon?

Not for months at a time, which is what I wanted to do (and did). Also, no, remote work wasn't really a thing at the time anyway. But more importantly I don't believe in being continuously employed your whole adult life. It feels dumb and unhappy.