A good case study is Dropbox. They didn't build any of their own infrastructure until they became big enough (2 years ago).
Primarily, until 2 years ago we did lean on S3 for all block storage, but most of the rest of the infrastructure (metadata storage, etc) ran in our own datacenters.
Your point I think you're getting at sounds like something I'd agree with though -- you can wait a bit the cost efficiency starts to be what is important/impactful to work on before shifting your usage away from some of these providers.
I don't know if it's true or not but I heard the story.