Them, "All that extra effort sounds great, but we don't have the time to do that. It will explode the really tight build, test, debug cycle we have now. Suddenly every cycle will be 100x as long."
Me, "First of all, you should be less concerned about the length of an individual cycle and more concerned about the sum of all the cycles up to completion."
Them, "Okay, but having short cycles also helps us course correct quickly, and keeps our 'velocity' up."
Me, "Well, second of all, what good is it to be able to iterate really quickly toward an unspecified goal? How do you know that what you're doing will get you there? Or worse, how do you know that what you're trying to do is even possible or not? Maybe you're 'engineering' your way asymptotically toward an impossibilty result."
It's sometimes crazy making to have these conversations and see how wildly distorted we've allowed our incentives and perceptions become as a group of professionals. It's become more important to produce the appearance of progress and effort than it is to actually make progress and thoughtfully apply effort.
I would also caution you. Do you really know the technology that you are trying to make safe? Building systems with research / deep learning is as different from traditional software development as synthetic biology from electrical engineering.
Yes, it's true that most companies hire way more people, way too quickly than is probably prudent for a problem domain (especially one that is still such an open-ended science project like AVs), and that as a result barely anybody has any idea what they're doing measured against what's being expected of them, but that's a wound that is entirely self-inflicted. It's not a natural law that one needs to suddenly staff up dozens or hundreds of really expensive people ahead of any real capacity to train or assimilate them, but it does happen with depressing regularity. If anything, I'd say this is probably the biggest problem I see in the ecosystem that you and I are in.
The team I work with is small. Everyone has several (5-15) years of experience consuming research, turning it into practical solutions, and in some cases even having created a novel practical application that was later supported by adjacent research.
It's not easy to construct a team of half-dozen to a dozen prolific programmers w/ a bent toward the academic, but an innate sensibility for remaining pragmatic and applicable. But, it's also not impossible to do so either. This is inline with my incentives comments elsewhere. I think the incentives are broken at too many layers to be able to be fastidious and intentional about how we solve problems. Say you get an "empire builder" into the leadership structure... you're doomed. The whole point of everything becomes about having as big of a "rockstar" team as possible, not about trying to solve a problem. Unless the problem one is trying to solve is how to quickly become the SVP of an extremely expensive and probably fairly unproductive team of increasingly disaffected previously very effective people.
That said, you'd be right. Not 15-years in something like computer vision. No such thing as formally verified NNs, regardless of what nuTonomy claims, exist. Even the basis about what one would be trying to verify is an open question. It's partially the point of DARPA's Assured Autonomy program.
The problem at the moment is that the ecosystem has two science projects on its hands. How to make an ML-derived driver is one. How to make a stable, correct substrate for acquiring data, make a decision, and react correctly is another. The only one most people hire up for is the former, and the latter is at least as challenging.
There's a layer of it (the AI/ML bits) which is necessarily probabilistic. That's by design. It's how it all works. All the layers below that today are currently terrifyingly probabilistic. It's difficult to advance the capability of the whole domain when the necessarily probabilistic parts are depending on shifting sands as a stable foundation.
I agree though that there's a large gap there in the domain expertise of the parties involved, and that's produced significant challenges as we've not yet figured out how to close that gap.
Though, I'm happy to be proven wrong here as it would make me a lot less nervous about the future were there a way to legitimately assure the artificial driver in an AV won't fail in emergent ways, or at all.
LUCRETIUS, De Rerum Natura
:)
I like to think of it in terms of quantifiers: for most systems it's fine to say that there exists a set of representative tests that the system passes. For critical systems, you really want "for all" in there instead.
Fortunately the model-based systems are much easier to reason about, even if they suffer from difficulty modeling certain cases. For a model-free system I think the starting point would be to precisely state a correctness criterion that could be used as a decision procedure on the output of the model; the combined system would be "verified" to a certain degree. Unfortunately you can't guarantee that it would always be able to yield an answer (it might always give a bad one which was rejected) and, the more fundamental problem, one might reasonably expect that if you can express the correctness property to a sufficient level of detail that you can perform verification on it you might be able to write an easier-to-verify implementation that doesn't rely on ML.
This stuff is hard, yo.
1. What parts of AI/ML have to be nondeterministic?
Once the model is trained then it should be completely deterministic. There are some training techniques that randomize the order or grouping of data, but they are primarily used because random is almost as good as finding the optimal ordering/grouping and finding the optimal ordering/grouping is hard. Sometimes randomness is used to augment the data to give you more examples to for against (because the model is too complex to train/fit with the data you have it you want to be robust to certain variations in input. If you're talking outputs then you should get the same outputs from a model given the same inputs. There are some really bad models out there that are way over fitted (relevant xkcd - https://m.xkcd.com/2048/ see "house of cards") and some people working on ML have a terrible grasp of statical techniques (just try models until they find one that fits), but even these models are deterministic.
What is so bad about nondeterminism anyway? Ethernet used nondeterminism to handle collision resolution. There best prime finding algorithms are nondeterministic. There is nothing wrong with appropriate nondeterminism.
It can't possibly be more efficient to take 50 fairly expensive people and have them iterate 10,000 times toward something that still ends up being fairly broken---and requires maintenance in perpetuity---than it is to take 6 slightly more expensive people, and have them iterate 10 times toward something that does exactly and only what it is supposed to do and requires basically no attention once it's been instantiated, can it? Even if you're Instagram.
That's why I'm sceptical about going to work on the autonomous vehicles tech (although it is tempting). I'm affraid there's been so much money and hype around it that "failure is not an option" - i.e. companies will sooner see another Challenger disaster than admit that they're not on track for their announced goals.