I have something similar. Not so much the colors...but I get a definite sense of being in different kind of "spaces" depending on the music. Hard to describe as there's no real visual aspect to it, but I might feel like I'm in a small tight room with lots of fabrics in it for one song, and under a huge concrete dome for another. I have certain "rooms" that I look for when I search for music I like.

I also frisson to music, sometimes very hard, it feels like raw lightening is being pumped through me.