Complex Car Software Becomes the Weak Spot Under the Hood
nytimes.com
nytimes.com
Across wide swaths of Fortune 500 companies, they abhor the idea of having tech people on staff (for reasons I will not go into here) and just now it's starting to bite them in the ass hard.
Proper discipline and established standards are, well, not established, so we get companies serving up product experiences to hundreds of millions of individuals where they are held together by duct tape and super glue (with varying degrees of quality but generally leaning towards overengineered+badly implemented).
That said, there are some standards in place for these devices, but regulation is lax and punitive action rarely taken (it's an honor system).
Compensatory and reactive costs are going to continue to rise as the steady flow of bad code borne from incompetence continues to accrue at exponential rates as more and more companies need tech built but aren't willing to expend the resources for it to be built properly.
Thankfully, as mentioned in the article, some auto companies are starting to embrace the idea of maintaining a healthy staff of developers in-house. Tesla, notably, has over-the-air updates pushed to vehicles directly rather than requiring owners to bring their cars to the dealer.
Tesla sold, what?.... 30000 cars all time? All models above $50000? So addressing people living in places where OTA is not an issue.
How are all the other car makers going to handle a production of a few milion units PER YEAR, being updated OTA? How are they going to update cars sold in countries where Internet is a dream? If you change your release strategy to fit an OTA update model, you better be able to support it. This is not a problem for the volume Tesla is selling, and neither for it's audience.
Ah, go on. I'm guessing it's partly a Two Cultures problem (not wanting to give tech people status) and partly a cost problem?
I always thought this was an engineering culture war thing -- software engineers aren't really engineers and software is sort of this silly thing that you use to glue together "real" engineering products.
Hopefully the author of your parent post will elaborate some :-)
(edit: I assumed you meant business-engineering culture war, not engineering-software. Might've been mistaken upon rereading)
When I was growing up in the 80's, my father was a software engineer at Honeywell and said they discriminated against the software guys, even when they knew software was the future of computing. He said it had been going on since the 60's and 70's as related to him by previous employees. He was told if he was a software engineer to just keep his head down and not talk to the "real engineers" working on "real problems".
Likewise when I was in college for computer science, he reminded me of the same struggles he had when I used to get fed shit from the other engineering students about how "If all else failed, they could become a CS major." and just be nice because one day they'll be coming to me for help.
And here we are today. Same thing, same arguments, same outdated viewpoints.
I mean to say, most people here will be highly familiar with version control, but back in the days when I used to audit companies, version control was barely[1] used, and editing files on the live server was the norm. And even here we're still having discussions about how "hard" it is to manage dependencies, use strong typing, use safe languages, garbage collection, bounds checking, formal proofs, etc.
[1] I think I may be correct in saying never used. I can't recall a case where we saw a company using any kind of VCS, but perhaps I just forgot.
I think it's a different problem. Software engineering is fundamentally about the management of complexity. People who have only worked on small programs -- like most non-software engineers and scientists -- don't understand the need to manage complexity because they've never worked on any program large enough to require it. Programming in the small is easy, particularly when you're mostly doing numerics. Other engineers see that and think that all programming must be easy.
Ideally you want something that is both deterministic (i.e., it's not running randomly in the background like a traditional GC) and that guarantees there are no leaks. In theory ARC gives you both of these.
In fact, I'd go so far as to say latency isn't the issue - the issue is that typically GC'd languages rely heavily on heap allocation because it's cheaper(!) not more expensive. Put another way: java creates lots of garbage, C/C++ does not.
If you wrote a C/C++ program that allocated as heavily as a typical java program... well, I'd expect performance issues that would probably be worse than java's.
> java creates lots of garbage, C/C++ does not.
Properly written C/C++ does not. Badly written C/C++ creates garbage (leaks memory), and it can never be recovered, because there's no GC. The solution isn't GC, though - it's to pay more attention to who owns the memory.
The ECUs were giving me frames with bits turned on at the wrong places. We used to call the telematics hardware/software provider which gives us an abstracted API to talk to ECUs quoting the problem. Their reply was "thats the ECUs messing up, we will "fix it" by giving you the bit turned on at the correct position or you change your code, because changing the API needs an OTA and OTA is not supported for that car manufacturer.
Except... anything that contains code has orders of magnitude more test dimensions than a typical physical part.
I would probably stop driving altogether if I knew exact figures on how many auto manufacturers even have in-house test coverage of their outsourced ECU / microcontrollers.
Suppliers also have tradition in car sub-systems. They also provide complete sub-systems (hardware and software). Much easier to implement the SW when you also build the full HW (injectors, pump, rail, sensors, ECU, etc.). You should see some of the development labs. Tens/hundreds of millions of EURO invested in equipment over time.
The only guys I know, who are still keeping a lot in-house, are the Japanese car makers (though they work with some locals in Japan, they can't be that independent). That market is opening a lot more to EU suppliers in next few years for sure.
In general car makers just lack the technological know-how to do a lot of stuff in-house. If they give up suppliers tomorrow, gonna take many years just to catch-up.
Does anyone know where this number comes from and if its realistic? It's hard for me to imagine unless you're counting all the LOC from Microsoft Windows: Car Edition or something.
Highly doubtful, Opportunity is at 4,000+ days of uptime...
This is simply not going to happen.
Firstly, the majority of automotive control systems on an ECU are running generated code from toolschains such as CATIA Systems, Dymola, MapleSim, OpenModelica, SCICOS, SimulationX, Vertex etc etc etc.
To validate against vulnerabilities requires in-depth knowledge of a particular manufacturer's modeling practices & standards. The code is not readable, the model is.
Which relates to the second reason, the IP in these models is very closely guarded. Competitive advantage and product differentiations are pretty much all software now. Even suppliers and manufacturers share black box models & systems back and forth for fear of leaking IP.
In a world where say BMW hands Bosch a set of requirement, Bosch supplies BMW HW and BMW validates in simulation and analysis with neither getting insight in to the other's design. You're telling that needs to now be done in the open?
I do not disagree with you. These are safety critical systems, I just don't see how companies could be forced to share IP.
What is more likely in my view is something similar to the validation standards in aviation: https://en.wikipedia.org/wiki/DO-178B
Most of the security breaches are due to people doing stuff really wrong not just a late patch. And even then I haven't heard of anyone dying as a result yet.
People have probably died (if you believe public health stats) as a result of increased NOx production of these vehicles. There's also been a lot more pollution. Big, big pollution problems have caused substantial changes to things, like the creation of the EPA here in the US.
Then people would stop using Windows, simple as that.
The people being harmed by software vulnerabilities are rarely the people who made an informed decision to use a certain piece of software. People who shop at Target can't meaningfully be said to make a choice to trust Target's software -- it's not something the average consumer things about.
So although "should" is an understandable prescription, "would" is a bit unrealistic...
Even speaking theoretically, the behavior you're describing isn't even an accurate description of how rational actors without resource constraints and with perfect knowledge would behave (because e.g. there are qualities aside from security and safety that might make it impossible to use an alternative product, and there might be qualities that are impossible to measure directly). And anyways, all three of those assumptions are clearly out-of-touch with reality.
Specifically wrt automobile software:
* Even with public access to source code, it's impossibly expensive for the average consumer (even the average software engineer; hell, even the average automotive engineer) to make a decision with truly perfect knowledge. Any number of safety constraints boil down to intractable questions if code is designed in a way that isn't amenable to inspection. So even in a purely theoretical setting with completely rational actors, making a fully informed decision might just not be possible.
* Many consumers will overlook errors in previous products and plausible errors in current products; Toyota's brand is doing just fine despite the findings of experts in the UA case. And I bet a lot of the software on cars that have sold even this year has some of the same structural problems, because based upon the expert testimony, a lot of that code probably needed to be essentially rewritten, and it's highly unlikely that they've achieved that in a year or two.
Is Linux more secure because everyone (quiet, Gentoo crowd) audits their own source? No, it's more secure because:
Anyone can choose to audit the source with minimal roadblocks.
Anyone who thinks they've found an issue can trumpet it from the rooftops.
Together, this means that even though I might not line-by-line audit my system, I can (with varying levels of paranoia / assurance) obtain a system that I have confidence others have audited. And furthermore, if there are issues found in the future then I can be reasonably sure of hearing about them.
Neither of these benefits is possible with closed source embedded systems.
(Or, from article "Ninety-nine percent of the buyers would never read anything. But out of the 11 million people whose car was cheating, one of them would have found it,” he said. “And Volkswagen would have been caught in 2009, not 2015.”)
My first point was exactly that transparency isn't a solution, even theoretically. There's no such thing as perfect knowledge when the questions you're asking are undecidable.
My second point was that even if transparency is a solution theoretically, it's demonstrably not a solution historically. People (including customers not subject to CYA-induced stupidity) choose insecure proprietary systems over secure open source systems all the time.
That said, something tells me that, in light of recent events, there will be a push for the automotive industry to start using things like DO-178, and its companions for mechanical and electronic engineering (whose numbers I don't recall).
Practically, air travel is safer because they are built, maintained, piloted, and continuously monitored by professionals. Conversely a 15 year old can build a kit car and drive it on the highway legally. If they break a dozen traffic laws they may or may not get caught and penalized.
If we could get people to assess and respond to risk rationally, we could make a lot of things safer.
Simply require source code publication for cars that are to be made street legal.
It's not going to happen unless legislated. That's the whole point. Currently there is no legislation forcing it to happen, but in the future there might be. That might be a good idea, even if the various companies involved don't want to do it.
1. The government told you to put spyware in e.g. your mobile baseband processor
2. Your GPU violates a lot of patents
3. You ECU implements dark patterns w.r.t. emissions testing
Make regulations requiring publishing the code, and require making the toolchain available to code can be verified vs. the shipped image. Proprietary software isn't a valid excuse when the stakes are to poison the environment.
No one is recommending copy/pasting aerospace regulations. In part because we're now 10-30 years further along in basic and applied CS research than we were when those regs were drafted.
What makes aerospace regulations expensive is the process, not the analysis. The attitude that formal methods are completely untenable is painfully sutck in the 1990's. For god's sake, Facebook runs portions of their code base through static analyses, as does Google. If websites are important enough to establish the safety of using formal methods, then the software running are ours sure as hell is.
In any case, thing is sure: as software becomes an integral part of high-level safety-critical control decisions, it needs to either be taken seriously by car companies and trusted by the general public.
Formal methods and static analysis are the only way we know of for establishing trust in software systems without revealing their source code or relying on black box testing (the downsides of which are demonstrated by both the Toyota UA and VW emissions testing cases). If you know of a third option, then by all means...
But going back to automotive, because that's what I do for a living. The process is quite strict and controlled. Efforts is done and periodic audit is done too to ensure adherence to ASPICE (http://vda-qmc.de/en/software-processes/automotive-spice/) and other regulations, including customer audits, which are tough. We do have internal processes, coding (and not only) methods, source control, configuration management (features, bug tracking) across all project and generic development base.
The basic workflow is follows a V-cycle path:
- receive requirements.
- requirements engineering, experts involved;
- test design based on requirements;
- internal design of whatever kind (models etc.);
- design reviews, simulations (if models), all kind of analysis;
- implementation (if not code generation);
- code reviews;
- static analysis ((pc-)lint, klocwork to name a few);
- module tests (unitary tests (ptu), tests on the HW with simulated signals - aka tetsbench tests);
- integration tests;
- system validation tests (requirements tests), hardware-in-the-loop tests, close loop tests;
- dependability tests run on the car.
In parallel the system is calibrated and calibrations are used during validations. For every version going into production (because some are just interim steps for validation purpose), there's a lot of testing going on on features already tested before.
SW execution is monitored by SW. The micro is monitored in some interesting ways and on top of that an IC is monitoring all the... other monitors. Lots of stuff is done to ensure RAM/ROM consistency, process execution monitoring, signal acquisition plausibility. Many inputs/executions diagnostics are requested by the laws, but a lot more are done for system robustness. Not gonna talk about HW design and validations. They have their own set of regulations and standard tests to fulfill, which I don't know that good.
This is where the car maker takes over from the supplier and does their own set of tests (which I don't know in much detail): - tens/hundreds of thousands of miles on a fleet of vehicles, recording all that happens and future analysis follows - cold trip, hot trip plus various trips to test response to different altitudes, sea side effects etc. - lots of garage testing - regulations tests (the car must respect various regulations of the countries it targets... OBD regulations, CARB, EU stuff, etc.)
A lot of other tests are carried out in different phases of production. For example a special SW which runs on the production line is designed to test all circuitry one last time before flashing the real working SW.
Too many people have the impression this SW is build "on our knees" (an expression that sounds good in my native language), but this is far beyond that. The threat of car recalls (which is, in the end, charged to the supplier causing it, or the car maker if it is his fault) and fucking up all the profit is not stuff engineering plays with. There are over 1k engineers working on engine ECU R&D in various locations (from system design to SW and tests).
Software should be required to be open source if the authorities that be can't police it (and they can't). Let the people do for themselves what the government can't. I can't think of a single, logical reason why it shouldn't be open source (profit is not a logical reason when lives are at stake). Until more people die, of course there will be no significant changes in this industry. In fact, according to history, until enough people die to reach the threshold for anger in the public at large, there will be no changes in this industry.
If it was, open source applications would never have critical bugs or security vulnerabilities.
It may mitigate some issues in a long run (e.g. intentional backdors) but in a short run it will create havoc as access to source code will make it easier to create exploits.
And suddenly you have whole new problem of making sure that everyone keeps their ECU software up to date or they risk fatal crash.
I'm not suggesting it is, simply that public oversight (open source) is the only way to ensure trust of a secure system. The system itself still has to be secure and requires QA like any other.
First you need to know the system, then you can understand what the code should do then you can have a fare chance of doing a code inspection.
I think people parallelize too much to web development or other environments a lot more exposed to public development. People think that if you can read someone else's C code for voice communication, they will also be able to understand the C code behind a diesel injection system, air intake path, exhaust gas treatment, fuel mass setpoints calculation. If you have never been exposed to stuff like that, you cannot fully judge if the code is wrong. You may find something obvious here and there (like a loop that may go beyond the limit in theory but never in the field).
And believe it or not, there is still competition for IP. Technical strategies of doing something in certain ways giving supplier X clear advantage in some field. Why would X share his knowledge just like that? We are talking about huge money. Never seen Google share the source of all they do.
Requirement creep seems to be the real problem here: Now we want aggressive fuel-efficiency targets, crash safety, and a host of other "nice to have" modern features. And it's in the computing unit which ends up trying to mediate these necessarily competing demands.