To add to your point: it doesn’t bother me so much that the AI system doesn’t share these priorities. What I worry about is that the companies designing them are not incentivized to.
every accident is a hit to their profits and pr.
It takes a hit if there isnt' a very solid marketing/pr/charismatic leader to distract
At the extreme end, picture a company that manages to get an okay from the city of SF for public testing, but whose internal tests show there is a small but significant risk of serious accidents and the product needs another two years of development to be safe. Their funding runs out in one year.
What do you think they would do?