We use a 'monorepo', so everytime we update any gem dependency, all of our apps get rebuilt entirely. So, updating all of our packages or not isn't a problem.
And I like the duplication, believe it or not. After having debugged environmental problems with dependencies in various languages over the years, I'm happiest knowing the dependencies in question are in vendor/bundle; end of story. Sometimes, in case of a nasty bug, I have a one-liner fix and I can go to vender/bundle, tweak the gem, and know I only affected the app using it. Those are things I prefer over some lost hard drive space.
Build times are a little long for us, though. That is true. I do a fresh bundle install on every build. That is the second longest part of the build, behind running our dog slow web tests.