Poorly-maintained roads are the problem that would give me nightmares if I worked on these. Lane markers that might have been perfectly usable when the database was compiled may be too worn out or obscured by snow, leaves, or who-knows-what else to be usable later. How can you design a system to fail safely when every input source is independently fallible? And how can we redesign the auto insurance industry to make it possible to try?
Google's self-driving cars still have trouble with heavy rain and snow (or at least they did as of last year). http://www.technologyreview.com/news/530276/hidden-obstacles...
I'd be curious to hear what the insurance industry makes of all of this too, they must be preparing for it in some way or another.