My first time deploying to Heroku in the summer of 2011 was a magical moment after spending a rather frustrating day futzing with a bunch of more barebones providers like Linode, AWS, Engine Yard, and few others I can't even remember at this point. I cannot stress how magical it was -- I like to think I'm a competent enough developer but I wasted a long time gluing things together and never quite getting it right for some reason. Before calling it quits for the day I decided to try that Heroku thing I had been reading and within 20 minutes I was up and running. That was impressive.
To note, I've been a happy paying customer of theirs since then, so they've made a decent chunk of change off my business over the last 6 years. If you can capture companies and projects in the infant stages, you can grow with them for quite a long time before it, if ever, becomes economical to roll your own infrastructure.
Currently working in a small startup and our data scientists (just a team of 6) are frequently fighting over the gpus in the office for time and online services atm are overpriced for our small budget.