I have the same issue, and with synaesthesia in general. In fact, my first (false) reaction is to disbelieve it when somebody tells me they have some form of it.

I am aware it's real and many people have synaesthesia but, I would guess, it's so far outside my perception of the world that my brain immediately wants to dismiss it. An interesting and somewhat annoying condition of my own brain.