We've also built and upgraded a few projects on Rails 4.1 recently and it has been very smooth.
On a side note: we avoid ActiveAdmin like the plague for the very reason that you specify here; our experience upgrading it between Rails versions has been terrible. We use our own simpler admin 'scaffolding' instead - https://github.com/SquareMill/generic_admin_controller/tree/...