The reason housing increases in value to the extent that it does is that areas where it's in high demand prohibit creating more of it. It would be like prohibiting the construction of new cars. Obviously that would make existing cars cost more in general, independent of the fact that some cars are worth more than others.
The problem with your argument is that it's predicated on it being physically impossible to build more housing of the same type that people want. If people prefer to live close enough to the ground that they don't have to ride an elevator for an hour to get there then you build a thousand ten story buildings instead of one ten thousand story building. If people like higher floors because they have a better view then you build two hundred 50 story buildings instead of a thousand ten story buildings. You can build more of whatever the market demands, so the price falls to the construction cost (absent zoning), and remains there because increased demand causes more construction rather than higher prices.
To get to the point of true scarcity due to land you would have to need thousand story buildings because covering the city in twenty story buildings wouldn't be enough housing for its population. But even if that's possible in theory, it's not the case anywhere in practice.