Hacker News Reader

Comment by chollida1 | Hacker News Reader

If Telsa and the article are telling the truth then these aren't anywhere near the same.

The mat was a design flaw from the beginning that missed QA, that happens in any large scale manufacturing as you can't just get everything right from the start.

If the article is telling the truth, this was a change made on the build line that wasnt' approved, that's a huge f$ck up if true and an incredible show of incompetence if someone can just start making design changes without approval on the build line.

View on HN

It’s better that a mat was designed in a dangerous way vs a production line mistake? That is similar to saying a simple bug is worse than an architectural flaw that no one caught at design time. Far more eyes are on the design flaw vs a production bug.

>vs a production line mistake?

I think the point they are getting at, if I understand the commenter correctly (and assuming the wording of the article is accurate), is that someone on the line had the ability to make a change to the production process without authorization.

That would not just be a "production line mistake", instead it is indicative of a serious policy and procedure failure. No single person on the production line should have the ability to make unauthorized changes to the procedures being used in production.

I hate analogies, but to use yours, it is a rogue employee that was able to change critical code with no approval process -- and no one else noticed that code was being changed and went ahead with shipping it out.

This is basically how all construction and manufacturing jobs work out, though? It isn't an isolated "single person" that can make arbitrary changes. They can propose something and it should be reviewed.

So, I don't think it is quite as simple as an isolated bug, per se. But it is very common for changes to get introduced at build time of physical things. Depending on where and what the change is, the level of review for it will be very different.

>This is basically how all construction and manufacturing jobs work out, though?

Not really. Any place with a decent QA department would sample a part, compare it to the specification, and raise an alarm because the part differs from the specification. There also should be occasional audits on the build process itself, which should have identified this, as it would differ from the specified process.

This type of issue (again, assuming the articles wording is true -- I have no idea) can only occur if there is either bad/missing QA, or bad/missing specifications.

>But it is very common for changes to get introduced at build time of physical things

Even in construction you need to have changes approved (i.e. a "change order" approved by the architect, engineer, and owner). Even extremely minor changes (which this would not be) must be documented on the "as-built" drawings.

> Even in construction you need to have changes approved (i.e. a "change order" approved by the architect, engineer, and owner). Even extremely minor changes (which this would not be) must be documented on the "as-built" drawings.

Do you really think this is what happens on job sites? Does this match your personal experience? Because my initial reaction was to laugh to myself at how rarely contractors, subcontractors, and crewmembers would actually engage a process like the one you are describing here. Non-spec stuff happens all the time without record, even in firms with solid QA.

>Do you really think this is what happens on job sites? Does this match your personal experience?

I worked in ICI (Industrial, Commercial, Institutional) construction for ~10 years. Yes, this matches my experience. Perhaps it is different where you are from.

I also experienced this while doing utility locating for oil & gas pipelines (~2 years). As-built drawings were very accurate, and detailed any deviation from the initial drawings.

In the 2000s I had a SaaS firm making software for underground utility locating companies so I learned a lot about the industry. In most parts of the country as built drawings are unusual for residential property anyway. Locating staff mostly shows up, looks at whatever drawings are available, and then has to figure out what was actually done from the clues and by using locating equipment. Many of these folks end up with a very subtle understanding of what common practice was by different utility companies in various specific areas in specific eras.

>In most parts of the country as built drawings are unusual for residential property anyway.

I was doing large transmission pipelines (i.e. NPS 24 to NPS 56), so I can't speak to residential, but I wouldn't be surprised if there was less attention paid to as-built drawings when the cost of damage/replacement wasn't in the millions of dollars.

I am involved with software that moves data between construction ERP systems and financial systems. Typically used in mid market commercial companies.

The single most commonly synced entity is Commitment Change Order items.

> Do you really think this is what happens on job sites? Does this match your personal experience?

Yes, it does. I’m a construction project manager, I’m not having my crew do any work that isn’t represented in the current revision of the plans and specs without approval because that’s the only way you get paid for the extras. Also if it’s an unapproved and unwanted change, you have to pay to remove it. Anyone managing a project who cares about managing their risk is going to submit RFIs and RFCs for every change.

It’s possible that the (tiny and insignificant) residential market is different, but that’s how commercial and industrial construction works.

It’s possible some tiny and insignificant changes like moving a receptacle or data opening a couple inches aren’t properly documented on the as-builts, but major changes almost always are.

> Because my initial reaction was to laugh to myself at how rarely contractors, subcontractors, and crewmembers would actually engage a process like the one you are describing here.

The firms you hire to work on your house aren’t representative of the firms who manage or work on commercial and industrial projects.

> It’s possible some tiny and insignificant changes like moving a receptacle or data opening a couple inches aren’t properly documented on the as-builts, but major changes almost always are.

Based on these responses I should have been more clear. These small and inconsequential things are what I'm referring to. Yes, the projects I'm familiar with track the medium and big stuff, and most of the small stuff.

I have seen multiple thousands of dollars of precast concrete get junked because an edge was less than half an inch out of tolerance. Multiple times. I have myself rejected multiple thousands of dollars of rebar because the hook length was short by less than an inch. Nothing that is shown in the plans or specs is inconsequential and payment doesn’t occur absent an approved variance.

All I’m saying is that out of spec work does happen. I’m not defending it.

and all I'm saying is that it doesn't make it on to the job in the first place usually and it certainly doesn't make it out to the public like with the cybertruck pedal

Absolutely! Why? Because it's your ass that's on the line should any of your "self-motivated" deviations cause financial harm, injury, or death, and you are going to be held responsible for those damages.

No one with any brains wants to be "that" guy.

That's why we have "cookie cutter" houses and even office buildings. All the kinks have been legitimately worked out and they can just crank them out. Bespoke construction? Cost overrun city. Now you know why.

This is going on the idea that there wasn't a documentation event with this change? I'm positing that knowing it is a recall on all of the trucks indicates that it was, in fact, a signed off change on the assembly line.

That is to say, just because it was on the assembly line doesn't mean it wasn't reviewed. And just because it was reviewed doesn't mean it wasn't a mistake. Part of the sign off was almost certainly "does not need retesting" for implementation. Which, was clearly a mistake. But isn't a sign of a broken QA system.

Relevant part of the quote which started this discussion:

>“[a]n unapproved change [...]

Unapproved, to me, implies that it was not reviewed or signed-off.

>Part of the sign off was almost certainly "does not need retesting" for implementation

If you are not assuring your quality, you have a QA failure. Regardless, I initially said "serious policy and procedure failure". Which, if you change a safety-critical component in your product and don't do testing on it, that is a serious policy failure.

Ah, totally fair. I took that to be "unapproved all the way back to the designer." Which, yeah, that doesn't happen. It almost certainly has approval from a line manager at the bare minimum, if it helps perform the assembly. If it goes to more teams than a single line, it gets more approval.

I think I largely biased to the next message, which did indicate reviews would happen, but that they have some freedom at the line. And that still sounds right to me.

This sounds like standard corporate ass covering to me. "Oh, that was just an unauthorised rogue employee, they've been fired" sounds a lot better than "someone suggested lubing up the accelerator to speed up production, and no one thought to check it won't cause problems".

For sure, I have no idea if the wording is truthful or just standard corporate blame dilution. But if the wording is truthful, this would be a significant process & policy failure.

If you ask me the lube just accelerated the problem. The root cause remains that you have a part secured with only a friction fit, in a setting where if that friction fit fails you have a a critical failure of the system. Friction fits can be very strong when properly established between appropriate materials, but this was not that. This was a cheap plastic cover made to be a bit too small over the metal lever. Over time with heat, sand/dirt, cold, pressure, vibration, etc. cycles, this was going to fall off regardless.

> It’s better that a mat was designed in a dangerous way vs a production line mistake? That is similar to saying a simple bug is worse than an architectural flaw that no one caught at design time. Far more eyes are on the design flaw vs a production bug.

:) I think you're missing my point, or I 've failed to explain it clearly.

A design flaw is bad, but we can't eliminate those. According to this article an assembly line employee went rouge and introduced a change without telling anyone.

If the article is correct then clearly these two things aren't even near comparable. We expect design flaws and adapt, we don't expect employees to go rouge and change the design without telling anyone.

Now the article or Tesla could be lying here but this is the facts as we know them.

Does that help clear things up for you?

I also dont' think you deserved the downvotes I saw you got for just misunderstanding. Sorry that happend to you!

I don't believe I misunderstand anything. This would be an interesting case study. It is very convenient to blame an employee "going rogue" for a dangerous issue like this. The design wasn't even changed. They just used a lubricant (soap?) to slide it on.

This overall points out the immaturity in Tesla's manufacturing process if changes like this can happen and then occur or affect every vehicle of a particular type produced, does it not? Overall, it still seems like a "below the line" change. These can still be quite impactful (see: memory corruption bugs leading to compromise and functional exploits). But it is still more akin to a bug or production flaw than a design flaw.

A design error leaves a papertrail for future study and redress.

An unapproved/undocumented production change may leave only the misproduced items. Mistakes happen, but this sounds more like changing the process without review.

Reply on news.ycombinator.com