Well. I'd exclude disabled people from this group of "nobody". For many people with disabilities a car is usually the only real way to get anywhere.
EDIT: I'm not saying wheelchair-bound and other similar people should be banned from driving. Like today, we can add wheelchair-only parking accommodations. In fact, restricting traffic to only people with a genuine need will make their trips faster.
Besides, even if you are on a wheelchair with lower body paralysis you can still drive with hand controls.