In my case I was quickly hitting limitations of Heroku, and the allocated resources. That said, you could hit the same (higher) limitations even if you're not using Heroku, and even if you're using physical servers.

My primary issues were:

- Lack of insight into CPU/IO/Memory usage

- Should have been able to handle much higher concurrency than I was

- Should have been able to perform much better than I was

The http://highscalability.com/ blog has a decent archive on the topic. Remember that your situation is unique and that the first rule of scaling is to measure everything you can. No two sites have the same audience, or requirements, or expectations. Things that work in one context will only create problems in another, and the context will change over time.

My own experiences tell me that the important things are:

1. understand http caching, and how to make it work for you.

2. query caching (whether against a database, a document store, search engine or what have you) buys you more than tweaking your web server.

3. your webserver configuration matters, but not as much as you'd think.

4. chunking a stream and sharding a datastore are two different views of the same problem.