Right now, many people have cars sitting idle for a large portion of the day because they need them for an hour a day, or sometimes an hour every couple of days, or a couple of hours a week. While it's true that there are surges in demand in the normal morning and evening commutes, there is still a lot of excess driving capacity that is needed for trips other than those surges, and so you only have to have enough cars to cover those peaks.
In addition, there may be other people who could take public transit, walk, or bike to work, but don't because there are other things that they need to do at various points that require a car, and so it's more convenient to just always have it available.
So yeah, driverless cars can only get you so far, you're right that the peak demand number is pretty much the minimum number you could have. But if I go an look at my residential street during the day, when commuters are presumably at work, I still see about half of the cars present. Those are all cars that wouldn't be necessary if everyone were riding in self driving cars that could rearrange themselves based on demand. That reduction in number of cars would also reduce the amount of space needed to store all of those cars, which could help bring everything closer together making it more walkable, bikeable, or efficient for public transit, which could further reduce the demand for cars.