I’d rather not have to manage a side account for AWS, and set up the connections to it and things. I’d rather it was just there, like their database.
Not trying to be a d*k or saying you are wrong, maybe I've only worked on simple-silly projects in my "medium'ish-long-career" so far but:
Why do you say that ? Really interested ?
If your resources are required, then they should be committed. If you want persistent cache, there are options both inside and outside Heroku.
I think I agree with you though - it's weird for me to have code on Heroku, and then log in to AWS, to put stuff in S3 manually.
It's if you want to store any kind of user uploaded or generated binary or large content - images, PDF reports, audio files, videos, JSON documents, code, logs - all common things you may want to manipulate in a web service.