Also, the average cost of raw, undeveloped land in 1970 was about $197/acre, or about $1690/acre in today's dollars. Today the average cost of raw, undeveloped land is about $4,350/acre.
All of the other costs associated with building a house are higher as well. There are more regulations, materials are more expensive, etc. Labor is probably the only part of the equation that isn't much different after adjusting for inflation.
So it's not just that we all decided to buy larger houses and that's why they are more expensive. There are a lot of structural issues at play.