It might be a bit because of the density, but housing is mainly cheaper near busy roads mainly because most people would prefer to live elsewhere, just like you.
That would explain a difference in price per square foot, but it doesn't explain why there are fewer small homes built further from busy roads. If anything you'd expect the opposite if it were just a reflection of people's preferences.
I think the unstated premise of that comment was that it's bad that we push the worst consequences of our societal choices onto the poorest people. Not that the mechanism by which we do it is mysterious.