I'm the same way. I never listen to music "casually" or in the background. All music is either something I'm really into, in which case I listen very actively, or something I don't care for, in which case I'd rather not be listening to it.
One potential exception that I've experimented with is classical music, but the high dynamic range (which is awesome) makes it difficult to listen to on headphones because parts are too loud and parts are too quiet.