There's not a hard line. There's a degradation in response time, and so old people drive slower (at least, they drive slower if they're wise). And then there's a point where even that isn't safe, and they need to stop.
But yes, not everyone can drive. Some are too old. Some are too young. Some can't afford a car for the amount of time they need to use one. Some are dealing with a temporary thing, like a broken leg or a bad sprain, and they can't drive just for a few days, but they still shouldn't drive right now. Some are coming back from a bar, or from a medical procedure involving anesthesia. (I forget the procedure, but I was told "don't sign any legal documents in the next 24 hours" along with "don't drive".) Lots of people can't drive for lots of reasons.
But, you know, we used to have things called "cabs" that helped with all of that. "Not everyone can drive" is orthogonal to "should a car be self-driving, or should a human drive it".