Lots of conditionals, yeah. This isn't really a bad thing, to be honest; it's far better than maintaining two separate branches. We already use similar conditionals when dealing with feature flags, so it's not really something that feels out of the ordinary in our app at all.
What's more, we found that if there was a file that had a ton of complicated, interwoven logic between the two versions of Rails, it was a clear sign we should see if we could instead backport a Rails 3 or 4 component so that we could nix the entire Rails 2.x-style functionality.