If all you're doing is hacking code then moving somewhere that lowers your burnrate as much as possible does seem like a good idea.
Also, if all your company is doing is "hacking code", then I doubt your company will be around for very long.
All that being said, I do agree with the sentiment, and I think that the benefits of high-cost locales are probably not worth the increased cost of living. I definitely appreciate my Midwest mortgage payment.
But some don't ever do in-person meetings. There are many small companies who do business 100% online, don't need investors, local staff or anything like that. For this type of companies staying in the big city makes no sense.
(Disclaimer: I live in the woods in a small mountain town)