The problem isn't so much the act of driving, though society would likely see some benefits from that time being used more productively (even if for leisure).
It's that the car is tied to you.
Everywhere you go, it needs a place to stay while you're doing anything outside your car. Parking spots average $5,000 per surface space, $20,000 per space in an above-ground parking structure, and $32,0000 underground. Every moment your car sits in one, it depreciates without performing useful work.