In that case you'd probably have a lot of empty trips, dropping the average occupancy of cars below 1. I think it's unlikely we'll see a lot of 2-4 person ride sharing (dislike of strangers, longer trip times due to needing to pick-up and drop-off more people). If you don't have higher density, the time for a vehicle to get to you will also be an issue.
It'd increase car-miles (likely increasing congestion) and likely lead to a move away from public transit in denser regions, akin to uber.
I'm highly doubful a large sharing economy for cars will happen anyway. Conventional car sharing (not uber) isn't popular and one reason for that is that people like having a personal car as a second "living room" (leaving personal belongings) without strangers and for the instant and hassle-free access it provides.
Thus I believe that self-driving will just lead to people driving more with their own personal vehicles, since it's becomes easier to do something else while driving and feels like less of a time sink.