A world where most households are man + woman + 0..3 children is very different than one where most households are 1..2 people + cat/dog. The latter demands far more housing than the former, even when populations are otherwise equal.
A population of 10 million singles demand 4 times more "housing units" than a population of 10 million people in 4-person families.
This is what makes immigration-driven population growth so pernicious compared to childbirth-driven. The family structures you end up with are completely different.