It sounds like it was tested on their own fleet but they accidentally pushed the wrong bits when deploying the update more widely out to customers.
Maybe they should have an additional phase between test deploy and customers such as "employee personal vehicles".
The usual "best practice" thing for IoT deploys, is to deploy to "your" devices, what for everyone to go green, then allow that build to deploy more widely. In a well-functioning system, it shouldn't be possible to swap bits between those stages.
But who knows what these guys were doing. :/
Reading between the lines of their public comments it sounds like they did run a full test through their test fleet, their employees, and then were rolling out to customers when the promotion process was "fat fingered". Maybe someone accidentally promoted the wrong release version.
Yes. And also things like rolling out the update in batches, and then also things like golden images, where if there are two crashes or failures in the first 24 hours of the update, change to the last known good software version.