Comment by ihsw | Hacker News Reader

Comment by ihsw | Hacker News Reader

ECUs have been notoriously badly programmed for years, they're usually contracted out rather than built in-house.

Across wide swaths of Fortune 500 companies, they abhor the idea of having tech people on staff (for reasons I will not go into here) and just now it's starting to bite them in the ass hard.

Proper discipline and established standards are, well, not established, so we get companies serving up product experiences to hundreds of millions of individuals where they are held together by duct tape and super glue (with varying degrees of quality but generally leaning towards overengineered+badly implemented).

That said, there are some standards in place for these devices, but regulation is lax and punitive action rarely taken (it's an honor system).

Compensatory and reactive costs are going to continue to rise as the steady flow of bad code borne from incompetence continues to accrue at exponential rates as more and more companies need tech built but aren't willing to expend the resources for it to be built properly.

Thankfully, as mentioned in the article, some auto companies are starting to embrace the idea of maintaining a healthy staff of developers in-house. Tesla, notably, has over-the-air updates pushed to vehicles directly rather than requiring owners to bring their cars to the dealer.

Yes you are absolutely right. The connected car project which I worked for had ECUs outsourced. The car manufacturer said "hey we are a car company". You deal with the technicalities of ECUs giving you proper data.

The ECUs were giving me frames with bits turned on at the wrong places. We used to call the telematics hardware/software provider which gives us an abstracted API to talk to ECUs quoting the problem. Their reply was "thats the ECUs messing up, we will "fix it" by giving you the bit turned on at the correct position or you change your code, because changing the API needs an OTA and OTA is not supported for that car manufacturer.

Which I think is symptomatic of the bigger issue. Car companies look at anything that goes into a car as a physical part, and they've gotten extraordinarily efficient (through ruthless competition) at process for dealing with physical parts.

Except... anything that contains code has orders of magnitude more test dimensions than a typical physical part.

I would probably stop driving altogether if I knew exact figures on how many auto manufacturers even have in-house test coverage of their outsourced ECU / microcontrollers.

abhor the idea of having tech people on staff (for reasons I will not go into here)

Ah, go on. I'm guessing it's partly a Two Cultures problem (not wanting to give tech people status) and partly a cost problem?

That doesn't make sense to me because car companies hire huge numbers of automotive engineers (I think?)

I always thought this was an engineering culture war thing -- software engineers aren't really engineers and software is sort of this silly thing that you use to glue together "real" engineering products.

Hopefully the author of your parent post will elaborate some :-)

(edit: I assumed you meant business-engineering culture war, not engineering-software. Might've been mistaken upon rereading)

The culture war is alive and well, many companies (mine included, unfortunately) treat software engineers as barely qualified typists of the "algorithms" designed by "real engineers". They'd get rid all of them in a second if they could, every so often a new manager tries.

The culture war has been going on forever.

When I was growing up in the 80's, my father was a software engineer at Honeywell and said they discriminated against the software guys, even when they knew software was the future of computing. He said it had been going on since the 60's and 70's as related to him by previous employees. He was told if he was a software engineer to just keep his head down and not talk to the "real engineers" working on "real problems".

Likewise when I was in college for computer science, he reminded me of the same struggles he had when I used to get fed shit from the other engineering students about how "If all else failed, they could become a CS major." and just be nice because one day they'll be coming to me for help.

And here we are today. Same thing, same arguments, same outdated viewpoints.

Well, unfortunately many software engineers don't behave like real engineers, so is it any surprise?

I mean to say, most people here will be highly familiar with version control, but back in the days when I used to audit companies, version control was barely[1] used, and editing files on the live server was the norm. And even here we're still having discussions about how "hard" it is to manage dependencies, use strong typing, use safe languages, garbage collection, bounds checking, formal proofs, etc.

[1] I think I may be correct in saying never used. I can't recall a case where we saw a company using any kind of VCS, but perhaps I just forgot.

Your criticism is valid from our perspective, but the non-software engineers on the other side of the culture divide are even less aware of proper software practice, so I don't see how the failure of software people to follow those practices could be the source of the prejudice.

I think it's a different problem. Software engineering is fundamentally about the management of complexity. People who have only worked on small programs -- like most non-software engineers and scientists -- don't understand the need to manage complexity because they've never worked on any program large enough to require it. Programming in the small is easy, particularly when you're mostly doing numerics. Other engineers see that and think that all programming must be easy.

Whenever I describe software engineering I typically use some form of "a true software engineer doesn't just make something work, he avoids &#%$ing over the next person to pick up his or her code."

Funny that you include garbage collection as part of what we should be doing, in an article about software in cars. Garbage collection is almost never the right answer for software in embedded systems.

Unless the poster was including ARC-style systems in what they meant by "garbage collection".

Ideally you want something that is both deterministic (i.e., it's not running randomly in the background like a traditional GC) and that guarantees there are no leaks. In theory ARC gives you both of these.

ARC depends on malloc/free which isn't quite as deterministic as you make it out to be - precisely because malloc/free does not (typically) run in the background, it needs to update internal datastructures when it gets the chance, which can lead to considerable latency spikes.

In fact, I'd go so far as to say latency isn't the issue - the issue is that typically GC'd languages rely heavily on heap allocation because it's cheaper(!) not more expensive. Put another way: java creates lots of garbage, C/C++ does not.

If you wrote a C/C++ program that allocated as heavily as a typical java program... well, I'd expect performance issues that would probably be worse than java's.

True, any malloc (or new) and any free (or delete) is non-deterministic (unless you're using a placement allocator). But GC isn't better, because the new still has the same non-determinism, Also, the GC has to do more work than a programmer-controlled delete (walking data structures for liveness, or something approximately equally expensive), and tends to do both the walking and the deletes in bursts. But...

> java creates lots of garbage, C/C++ does not.

Properly written C/C++ does not. Badly written C/C++ creates garbage (leaks memory), and it can never be recovered, because there's no GC. The solution isn't GC, though - it's to pay more attention to who owns the memory.

What makes you believe in-house SW is more reliable? Anecdotal or not, I know people who worked for both a X's (German car maker, not VW) supplier and X themselves in SW engineering and he told me the supplier was on a totally another level above the in-house development process.

Suppliers also have tradition in car sub-systems. They also provide complete sub-systems (hardware and software). Much easier to implement the SW when you also build the full HW (injectors, pump, rail, sensors, ECU, etc.). You should see some of the development labs. Tens/hundreds of millions of EURO invested in equipment over time.

The only guys I know, who are still keeping a lot in-house, are the Japanese car makers (though they work with some locals in Japan, they can't be that independent). That market is opening a lot more to EU suppliers in next few years for sure.

In general car makers just lack the technological know-how to do a lot of stuff in-house. If they give up suppliers tomorrow, gonna take many years just to catch-up.

over-the-air updates for safety-critical components of a car is a risky practice at best.

Having worked in exactly the same domain , I can very well say that OTA is risky, moreover the data providers for the car is not really happy about OTA ( Sprint , AT& T provides connectivity for the projects I worked on.). They lease out a certain bandwidth to the connected cars which sends small bursts of data at a low contracted price. If the car manufacturers need OTA, they will have to pay a premium which will be slapped on the customer. There is also a technicality of downloading a huge image ( possibly 1 GB ) over a spotty network. ( a running car will switch cell networks , network types 3G , LTE , 1x , lost connections etc ).

Why only Tesla has OTA? Some people really think nobody in the industry ever thought of that since the internet was born? Musk is doing a nice publicity stunt.

Tesla sold, what?.... 30000 cars all time? All models above $50000? So addressing people living in places where OTA is not an issue.

How are all the other car makers going to handle a production of a few milion units PER YEAR, being updated OTA? How are they going to update cars sold in countries where Internet is a dream? If you change your release strategy to fit an OTA update model, you better be able to support it. This is not a problem for the volume Tesla is selling, and neither for it's audience.

Reply on news.ycombinator.com