It doesn't cost that much. The land costs a lot, and permits can cost a lot. The labor isn't that much, and materials are cheaper than ever (though most houses in the past were built with free labor with material prepared on site and so you don't realize how much it costs as nobody was getting paid)
The labor is low because it's minimum wage based for hard work (where I am at in Texas, it's basically all foreign labor, legal and not). Paid a respectable wage, this labor would be expensive. But it's exploited.
Building a stick frame house on site is the cheapest out of all the methods. The expensive parts are the finishes and appliances.
Also, houses could easily be designed in a simpler fashion (a rectangle without any modulation, ductless HVAC, etc.) for an inexpensive build.