Stereotypes don’t materialize out of thin air. If they did that you’d see a more random distribution of competing stereotypes.
After all, if you hold the belief that women are bad drivers, did you independently come to this conclusion based on an accurate assessment of driving skill and gender, or did you hear "ha, women drivers!" over and over as a kid and subsequently see every mistake a woman driver made and mutter to yourself, "ha, women drivers!"?
When you label something as confirmation bias, make sure you're looking through the right end of the telescope.