I have my homedir in git too and also ignore just about everything. I ended up using gitignore as a whitelist instead of a blacklist, e.g.:

    /*
    !/.gitignore
    !/.gitmodules
    !/.gitconfig
    !/.zshrc
    !/.zshenv
    !/.zsh
    !/.tmux.conf
    !/.vimrc
    !/.vim