I've upgraded several apps, I've been on the Rails train since 1.2, and each time there's been slight, and more often major, performance gains. Yes, some of these apps where big (Not Twitter big, but big enough. And no, they didn't need to do anything too crazy hardware wise.)

Almost all Rails apps that are slow on performance suffer from architectural flaws or oddities in the code. A lot of time Rails get a beating when it boils down to slow, illformed SQL queries. One time I had a customer who wanted me to rewrite their platform because "Rails was so slow", when in fact they did some heavy image processing and uploading to a cloud storage within the browser request. I'm not saying OP does any of this, but since I don't know any details about their codebase or infrastructure I'm making some general remarks that may or may not apply.

Ruby code can be written in so many ways, I guess that is its blessing and its curse. There are bad ways to do things, and there are good ways. With great freedom comes great responsibility.

I would encourage OP to push on and upgrade to Rails 4.0.1. The gap between Rails 3 and Rails 4 isn't that huge at all. I'm sure you'll be pleased.