Dear Everyone Doing Devops: do not build gems on your production servers. Build a tarball, test the tarball through dev/staging, deploy the tarball to production.
For any value of tarball- literal tarball, deb/rpm (actually CPIO under the hood, but who cares), prebaked AMI (not tar at all), docker image (tarball of JSON + nested tarballs. Seriously, it's tar all the way down).
Everything will be better. Roll forwards. Roll backs. Speed. Attack surface. Reliance on 3rd party services that will rate limit you hard.