It's quite impressive it kept working for that long, and even more amazing someone was capable of saying "no, let's not cut the power, there might be a problem if we do"
Microchips, capacitors, PCB's and other stuff like manufactured for defence have much wider and higher tolerances. One won't find the cutting edge 2 nm processes (depending on the specific area of defence, 200-300 µm would still be cutting edge today for them), materials used are different, plus all sorts of chemical and physical hardening is employed. Most of the time, the chips can't use the consumer grade cutting edge designs and processes as they have to withstand not just abuse but also stronger EM fields (in radar equipment) or EM pulses (e.g. a nuclear explosion).
After electronics have been manufactured, they proceed onto the QA phase where chips are literally baked in ovens for a day or a few, then retrieved and tested. They may get irradiated, soaked for periods of time etc. Most people would call it abuse – from a civilian perspective. The surviving ones are deemed to have passed the QA and are used to make the actual equipment.
Sorry, I think you're off by a few orders of magnitude, do you mean 200-300nm?
Smaller processes result in chips either becoming very unstable or plainly kicking the bucket when exposed to strong electromagnetic fields and electromagnetic radiation. Radar equipment generates such strong EM radiation that is lethal to humans and to microelectronics no matter how well shielded the chips are. Then there are highly charged particles released during a nuclear explosion, so the electronics have to survive at least for some time at the battlefield.
Chips produced for defence and military use some of the most conservative design and manufacturing processes, and even the analogue processes may still be used somewhere (I have not checked recently though).
Some people are so terrified of having to say "I was wrong" that they either will do nothing, or just happily follow decisions that someone else makes. It's one thing when that's a line employee behaving that way, but there are managers so terrified of making a mistake that all they really are is scribes, tracking who said what and who decided what. These weasels don't lead. They don't contribute anything. They just have a quick answer when someone above them asks how this fuckup happened. Well that was Jameson. He decided to do X so clearly it's his fault things broke.
I was in a room once with a guy who was so caught up in the idea of catching me in a gotcha that he had no idea what to do when I said, "yep, that's wrong, and this is how we fix it." His face went through the same roller coaster of emotions that happens when you drop your ice cream cone. I got to say most of what I needed to say before he even recovered. I think I was proudest of keeping a straight face during the entire conversation.
Where has intellectual honesty gone that just being transparent verges on a superpower? I feel like I'm taking crazy pills.
I had your view until 7 years at Google. Showing this to 80% of people there would garner tsktsking and quips about delusions of grandeur and seeming unbalanced. In performance reviews, you could count on this person saying you collaborate poorly and don't take others concerns into account, because they felt humiliated and retconned a reason why it's your fault. (To be fair, given your vivid understanding of their face at the time, it sounds like you understand they wouldn't feel well after you did this)
In the higher rooms, I saw at least a couple times everyone living this worldview and it ending up in bickering and extreme insistence the other person was missing fundamentals while nothing got done for months: which in the more Machiavellian parlance, is still good if you're good, because most people won't do the work, just argue over the 20% of the problem they understand, and someone has to win eventually
I keep editing this comment because I get the vibe it disagrees with you, but it doesn't, at all. If anything, your framing helps sharpen why it felt like an odd place, in a bad way, to me from day 30.
Sadly, this culture must have deteriorated; my friends who are still at Google say that it's become literally the stuff of Dilbert comic strips :-(
But I'm not.
I did Aug 2016-2023. Of course it's a complicated story, and you've heard a lot already I'm sure, but when you mention blameless postmortems, it remind me of my pet peeve: promo committees done within teams (basically 2 levels up, i.e. skip is doling out promo, with no recourse or real outside input) was a horrible, wretched idea.
There was this odd obsession with saving time on perf, but they kinda just needed to tell people to chill and recognize that it really is important for management. With all promos being judged by people next to you, what we might call "being agreeable" became a necessity.
I distinctly remember in 2018 being told if things were how they had been, that I should just do a blameless postmortem, but with things as they are, that was a quick way to end up on the bench for the rest of my career.
Looking back, IMO the seeds of its downfall had already been planted by the time I was there, including the culture of blameless postmortems.
There was this idea that any problem—any outage or whatever —reflected not stupidity on the part of the perpetrator (Google hires the smartest people after all) but a process failure. Blameless postmortems avoided blame by ending with a process recommendation.
I’ve worked at startups since, and I do think that creating an environment where people are comfortable taking risks is just a fundamentally hard culture problem, and you don’t even have to be big to have it (though it doesn’t help). I’ve seen 15-person startups where engineers wouldn’t fix something because “I’m just an engineer and this is really a product decision” while the PM says “we need to get the engineers together in a meeting and come to a consensus on this.”
IMO Google’s hack of giving people space to take risks by letting them pass off failure as a process problem did sort of work, to their credit, until they got to where it sounds like they are now, where you can’t do anything without being accused of bypassing “good process,” which means going to endless lengths to get everyone’s opinions and placating everyone who disagrees with you.
It seems like at Meta they solved the problem by regarding failure as good and just. At my current company, to do something risky, you have to write a design doc and have a comment period, and then you’re free to do whatever the doc proposes. Idk what other approaches there are—I’d pay to read a book about it, though.
To make the problem worse, nobody dares to remove from a process as that would mean they would be accountable for the consequences of the "missing steps", and at the same time adding to a process is almost always safe - another step in a process to clarify or verify can't really fail and having contributed to the process work in a process driven organization looks fantastic on your yearly review. So we get more and more processes that are perpetually growing in scope.
The long term damage that heavy processes do, what I've seen first hand, is extreme stagnation. Even simple and straight forward improvements are avoided if it means that you need to produce ridiculous documentation that nobody will ever need, or seek approval from a (very busy) group of people who contribute nothing other than checking that you followed the processes.
Shorter processes in clock time matter more than step count, but it’s hard to do one without the other.
But the other one is that blameless doesn’t mean we don’t talk about the person who fucked up. The entire point of the NTSB’s version of the process is to get pilots and ground crew to be vulnerable. To explain how they pushed the wrong button or signed off on a maintenance deferment or whatever. I was tired, I got asked tangential questions during a critical moment. I misinterpreted the text, etc.
I’m neurodivergent. I spend way too much of my day analyzing my errors. But the way my brain works on a Tuesday is the way everyone else’s works in the middle of a high stress situation. For me it still feels like a bad Tuesday. The tools and steps I build for keeping me from looking stupid also work for the team, once they give them a chance. And during Five Why’s I just have to say the quiet parts out loud, and I can speculate rather well on whether other people had similar experiences.
Your average person doesn’t even take authority.
The next “level” per se (not that you have to take authority — only if you want to) is the person taking authority.
Then the next level above that is the person that takes authority but understands how it looks to other people, so they sure everyone else understands that they are taking authority to make things move along, not because they are entitled to it. They attack the problem head on.
My goal is to get good engineering done while also making sure everyone feels included and good about being at work.
If you’re about good engineering and respect for other people, you can get things done however you want.
If you don’t act like a lead dev you’re not going to get much practice at being one. And I’m not talking about the bossing people around part, but the fostering teamwork and taking care of people parts.
I’m not saying own all the decisions. With a largish team you want at least three people who will stand by a decision even when it’s not a unanimous vote. (Don’t ever do four or six unless you have a good boss. Jesus Christ that was awful.) When I do say something will definitely work, I’m almost always right.
What I left out and thought of later is that it’s more important to own the cleanup than the decision. Once you do that people trust you more, and the trust leads to other opportunities. So it’s not really, “I was wrong” although that already filters out way too many people. It’s the “here let me help you fix it.” That makes it stick.
The handbook or SOP somewhere probably does call for a risk assessment.
So they have two choices. Choice A is to do what's obvious, cut power and get to work. That has two branches. One branch is saving the day and potentially millions of dollars. For what? A side mention at the next all hands, maybe. Branch 2 is something goes wrong, and the company says you didn't follow the handbook, and you're fired or worse, found liable.
Choice B is following the handbook, knowing everything will be ruined. But, you keep your job, and maybe the handbook gets updated.
Both choices provide no reward. Only one provides no risk.