I do think that when you are first launching a startup, you should probably do everything as cheaply as possible, which probably means using a VPS or IaaS. As soon as your infrastructure becomes more complicated, you can use Heroku instead of hiring people to manage it for a while. Then see where your growth takes you -- you will probably find that eventually it makes more sense to hire a couple people to manage some IaaS or even renting racks and buying hardware.
As with all things in life, it really depends. I believe this particular configuration makes sense for companies who want to focus on product, not infrastructure -- at a cost of expensive hosting.