The big problem with bundle exec (or Ruby startup times in general), is the ludicrous amount filesystem access because of searching through a ridiculous large number of files for each "require". E.g. if you have a ton of gem's installed, most "require" calls will look for the files you require relative to the root of every gem....
Much more extensive use of require_relative, and fewer search paths can fix that entirely.
Try an "strace 2>&1 -e last64,stat64,lstat,stat bundle exec [yourcommand] | less" and be prepared to be shocked at the waste.
(EDIT: This of course assumes you have strace; on most Linux distro's that's just a package install away - I don't know about OS X, and I've got no idea how to make dtrace do the same)