I think there's an inherent distrust (which I share) of giving technology the power to end our lives with one bug. Trust in technology, in general, is pretty low right now, so that reflects back on perceived future technology where one bad steer or missed brake ruins your life. Although I happen to share this pessimism with the human aspect of driving as well, I've almost died before because I wasn't focused while driving (IRL nerd sniping basically), and I know it could happen again in the future no matter how much I want to be a safe driver.
The truth is, if full self driving cars actually happened, with safety statistics better than humans and with a low barrier to entry, you can bet that people would change their opinion real fast. Much like Americans who have a revelation after experiencing European or Asian public transit, the realization that they could have an extra hour per day that was previously spent staring at a road, would be a big motivator to change opinions on technology. Add in that three of your friends already use it daily and they're not dead yet, and you rode with them and it seemed like a safe driver.