Hacker News Reader

Comment by ziddoap | Hacker News Reader

>vs a production line mistake?

I think the point they are getting at, if I understand the commenter correctly (and assuming the wording of the article is accurate), is that someone on the line had the ability to make a change to the production process without authorization.

That would not just be a "production line mistake", instead it is indicative of a serious policy and procedure failure. No single person on the production line should have the ability to make unauthorized changes to the procedures being used in production.

I hate analogies, but to use yours, it is a rogue employee that was able to change critical code with no approval process -- and no one else noticed that code was being changed and went ahead with shipping it out.

View on HN

This is basically how all construction and manufacturing jobs work out, though? It isn't an isolated "single person" that can make arbitrary changes. They can propose something and it should be reviewed.

So, I don't think it is quite as simple as an isolated bug, per se. But it is very common for changes to get introduced at build time of physical things. Depending on where and what the change is, the level of review for it will be very different.

>This is basically how all construction and manufacturing jobs work out, though?

Not really. Any place with a decent QA department would sample a part, compare it to the specification, and raise an alarm because the part differs from the specification. There also should be occasional audits on the build process itself, which should have identified this, as it would differ from the specified process.

This type of issue (again, assuming the articles wording is true -- I have no idea) can only occur if there is either bad/missing QA, or bad/missing specifications.

>But it is very common for changes to get introduced at build time of physical things

Even in construction you need to have changes approved (i.e. a "change order" approved by the architect, engineer, and owner). Even extremely minor changes (which this would not be) must be documented on the "as-built" drawings.

This is going on the idea that there wasn't a documentation event with this change? I'm positing that knowing it is a recall on all of the trucks indicates that it was, in fact, a signed off change on the assembly line.

That is to say, just because it was on the assembly line doesn't mean it wasn't reviewed. And just because it was reviewed doesn't mean it wasn't a mistake. Part of the sign off was almost certainly "does not need retesting" for implementation. Which, was clearly a mistake. But isn't a sign of a broken QA system.

Relevant part of the quote which started this discussion:

>“[a]n unapproved change [...]

Unapproved, to me, implies that it was not reviewed or signed-off.

>Part of the sign off was almost certainly "does not need retesting" for implementation

If you are not assuring your quality, you have a QA failure. Regardless, I initially said "serious policy and procedure failure". Which, if you change a safety-critical component in your product and don't do testing on it, that is a serious policy failure.

Ah, totally fair. I took that to be "unapproved all the way back to the designer." Which, yeah, that doesn't happen. It almost certainly has approval from a line manager at the bare minimum, if it helps perform the assembly. If it goes to more teams than a single line, it gets more approval.

I think I largely biased to the next message, which did indicate reviews would happen, but that they have some freedom at the line. And that still sounds right to me.

> Even in construction you need to have changes approved (i.e. a "change order" approved by the architect, engineer, and owner). Even extremely minor changes (which this would not be) must be documented on the "as-built" drawings.

Do you really think this is what happens on job sites? Does this match your personal experience? Because my initial reaction was to laugh to myself at how rarely contractors, subcontractors, and crewmembers would actually engage a process like the one you are describing here. Non-spec stuff happens all the time without record, even in firms with solid QA.

I am involved with software that moves data between construction ERP systems and financial systems. Typically used in mid market commercial companies.

The single most commonly synced entity is Commitment Change Order items.

> Do you really think this is what happens on job sites? Does this match your personal experience?

Yes, it does. I’m a construction project manager, I’m not having my crew do any work that isn’t represented in the current revision of the plans and specs without approval because that’s the only way you get paid for the extras. Also if it’s an unapproved and unwanted change, you have to pay to remove it. Anyone managing a project who cares about managing their risk is going to submit RFIs and RFCs for every change.

It’s possible that the (tiny and insignificant) residential market is different, but that’s how commercial and industrial construction works.

It’s possible some tiny and insignificant changes like moving a receptacle or data opening a couple inches aren’t properly documented on the as-builts, but major changes almost always are.

> Because my initial reaction was to laugh to myself at how rarely contractors, subcontractors, and crewmembers would actually engage a process like the one you are describing here.

The firms you hire to work on your house aren’t representative of the firms who manage or work on commercial and industrial projects.

> It’s possible some tiny and insignificant changes like moving a receptacle or data opening a couple inches aren’t properly documented on the as-builts, but major changes almost always are.

Based on these responses I should have been more clear. These small and inconsequential things are what I'm referring to. Yes, the projects I'm familiar with track the medium and big stuff, and most of the small stuff.

I have seen multiple thousands of dollars of precast concrete get junked because an edge was less than half an inch out of tolerance. Multiple times. I have myself rejected multiple thousands of dollars of rebar because the hook length was short by less than an inch. Nothing that is shown in the plans or specs is inconsequential and payment doesn’t occur absent an approved variance.

All I’m saying is that out of spec work does happen. I’m not defending it.

and all I'm saying is that it doesn't make it on to the job in the first place usually and it certainly doesn't make it out to the public like with the cybertruck pedal

>Do you really think this is what happens on job sites? Does this match your personal experience?

I worked in ICI (Industrial, Commercial, Institutional) construction for ~10 years. Yes, this matches my experience. Perhaps it is different where you are from.

I also experienced this while doing utility locating for oil & gas pipelines (~2 years). As-built drawings were very accurate, and detailed any deviation from the initial drawings.

In the 2000s I had a SaaS firm making software for underground utility locating companies so I learned a lot about the industry. In most parts of the country as built drawings are unusual for residential property anyway. Locating staff mostly shows up, looks at whatever drawings are available, and then has to figure out what was actually done from the clues and by using locating equipment. Many of these folks end up with a very subtle understanding of what common practice was by different utility companies in various specific areas in specific eras.

>In most parts of the country as built drawings are unusual for residential property anyway.

I was doing large transmission pipelines (i.e. NPS 24 to NPS 56), so I can't speak to residential, but I wouldn't be surprised if there was less attention paid to as-built drawings when the cost of damage/replacement wasn't in the millions of dollars.

Absolutely! Why? Because it's your ass that's on the line should any of your "self-motivated" deviations cause financial harm, injury, or death, and you are going to be held responsible for those damages.

No one with any brains wants to be "that" guy.

That's why we have "cookie cutter" houses and even office buildings. All the kinks have been legitimately worked out and they can just crank them out. Bespoke construction? Cost overrun city. Now you know why.

This sounds like standard corporate ass covering to me. "Oh, that was just an unauthorised rogue employee, they've been fired" sounds a lot better than "someone suggested lubing up the accelerator to speed up production, and no one thought to check it won't cause problems".

If you ask me the lube just accelerated the problem. The root cause remains that you have a part secured with only a friction fit, in a setting where if that friction fit fails you have a a critical failure of the system. Friction fits can be very strong when properly established between appropriate materials, but this was not that. This was a cheap plastic cover made to be a bit too small over the metal lever. Over time with heat, sand/dirt, cold, pressure, vibration, etc. cycles, this was going to fall off regardless.

For sure, I have no idea if the wording is truthful or just standard corporate blame dilution. But if the wording is truthful, this would be a significant process & policy failure.

Reply on news.ycombinator.com