I've tried this and a lot of other dotfile managers (homesick, homeshick, etc), my current favorite and what I'm using now is YADM:
For example, if you want to keep track of your whole Firefox profile's directory (without bothering adding/committing single files), in stow it's as simple as symlinking ~/.mozilla/firefox/<profile_dir>.
For programs whose configuration you know intimately this is not a problem (e.g. for bash, we all know we probably just need .bashrc), but not all programs rely on a single flat configuration file.
I have a git project called .linux-env. I can just clone it on any machine and ln my ~/.bashrc/~/.profile to it. Even works on mac.
unamestr=`uname`
if [[ "$unamestr" == 'Linux' ]]; then
source ~/.my_environment/bash/linux/.bashrc
else
source ~/.my_environment/bash/osx/.bashrc
fi
etc.