The issue is overpopulation. In an underpopulated area neighbors tend to be neighbors, in an overpopulated one they tend to call police on your noisy kids to take you down a peg.
It's actually about restrictive zoning preventing sufficiently dense housing and high quality public transit.