You should look at something more along the lines of rdiff-backup (or whatever's fashionable these days) for that.

Using git would be pretty pointless, unless you're going to be frequently branching and merging against your mp3s or something.