I've done plenty of computer vision projects and pitched in on some autonomous driving/flying hobby projects- so I'm definitely willing to accept our robot driving overlords, but maybe I know just enough about it to know we're not all the way there yet.
I feel like Tesla is in a bit of a Catch-22 in that to get the system better you want as many people driving with it and feeding you data. At the same time people are people and they will get lulled into a complacent state and begin not paying attention pretty quickly.
That being said, if there was some data that said right now, today, Tesla autopilot is better than the average (or maybe up to the 75th percentile) driver, I'd be okay with people just letting it take the wheel and not screwing things up.
EDIT: Just to be clear, we were in a Tesla w/ autopilot. We weren't just coasting down the highway at rush hour in a regular car.