dedicated hardware. common. that might have been an option in the 90s. i want to be able to use as many cpus as i want when i need them. not have a bunch of machines rotting away idle.
For our service we are going to use a co-lo server for our processing with an Elastic Beanstalk frontend and use a VPC + OpenVPN setup to bridge the two. We will incur some bandwidth charges because of this, but the cross talk between the boxes is actually minimal, since the client will post directly to the co-lo box(es) when it needs to upload, etc.
Amazon is a huge premium over physical hardware. You use them when a) you want to scale immediately, b) you have financial reasons for not buying your own gear for long-term use (1-3 years), and c) you don't mind paying the premium.
It's really cheap to try and see if it's good enough for you.
ass.