Not considering each neighborhood is how we get here. If you treat the city as a whole as a community, you lose the granularity and it ends up as you describe - the rich veto in their area.
But what is your solution? The section 8 housing or other universally unpopular projects have go somewhere. Do you think the SFH owners in the poor part of town want it in their backyard? It has to go somewhere. It's not about containing people like toxic waste. It's about existing individuals not wanting to deal with a reduction in their estavlished quality of life over statistically founded concerns (not appearence related but project specific, such as traffc fatalies, noise, etc).
"People want a single family home but is it because they want the shape or because society is arranged such that it is functionally the only way for the middle class to build wealth and feel a sense of housing stability?"
Does it matter? Is there a solution that would make a difference? For most people, this isn't really building wealth because you need to live somewhere, so that wealth isn't available to use. From my experiences and talking with others, it is mostly because of the stability, freedom, and not having to deal with bad neighbors as closely. I'm not sure what costs you are referring to that wouldn't apply to other types of housing. The only real subsidy homeowners get would be the property tax deduction. Maybe some developers worked out deals that indirectly benefited homeowners, but so too for the apartments. Where I'm at, you have to pay to install utilities from the street, pay taxes to support the infrastructure, etc. In fact, SFHs tend to pay more taxes than condos and apartments, subsidizing the schools and other property funded programs.