People forget that the average home was ~700 square feet 100 years ago. It's 2400 today. How many people would be willing to put a family of 4 or 5 in a 700 square foot home today? Go to old towns and look at some of the houses that are 100+ years old and 2400 square feet. You know who lived in those homes 100 years ago? Really rich people like lawyers and business execs and they had servants cooking and cleaning. Today that's just expected to be a home for a normal family.
We could change what we think a home is for most people but I don't think most people are willing to accept that the home they can afford is closer to a prefabricated trailer than it is a traditional single family home.
Even apartments are super expensive to build and the reason you don't see many dwellings that aren't "luxury" is because the cost to construct and maintain them is so high that you'd never get your money back otherwise.