Speaking as both a provider and someone who has managed many servers for other people, I would argue that you are best off if you base your system on something that is commodity... that is, don't buy something from someone else unless you can get the same product from two vendors. Locking yourself in to one vendor is nearly always a bad idea.
That said, Personally, I would suggest you go for what you call an 'infrastructure platform' - if you are just renting [virtual] linux boxes, you can rent those from many providers. Once you start needing a larger box, buying your own hardware will save you a whole lot of money over something like ec2. (though, ec2 is pretty nice for scaling quickly)
When just starting out, well, personally I'd suggest a small VPS, though I have something of a vested interest in the matter. I've heard nothing but good things about Linode, and their prices appear reasonable. If you are willing to accept something that has a much more 'raw' interface, you can check out my offerings, see the link in my profile. My prices are
But either way... the important part is to avoid that lock-in. Companies change. Sometimes a company that was previously reliable makes some bad decisions and becomes unreliable. Other times, like in the case of EC2, some companies start with really good prices, but fail to lower them as the price of infrastructure falls.