I do something similar - mostly because buildpack downloads seen to time out ever second time. Just wish I could find a trivial way to cache installed gems to really get things flying along.
Bumping the patch version on your CSS library resulting in a full redownload and recompile of every gem sucks.