The problem of qualia comes into play most especially when a materialist metaphysical position is accepted of the kind where the immaterial res cogitans is denied. The materialist is then faced with the task of explaining how all of that which was attributed to minds (e.g., color) -- and indeed minds themselves -- is really something that can be attributed to res extensa. The kicker is, of course, that you can't, at least not without radically reforming how you understand matter. It is by definition impossible to locate color in matter given a Cartesian understanding of matter. Many physicists, in their more philosophical moments, tacitly accept a metaphysical position of that kind.
What's important is that the modern understanding of color, one that attributes it to consciousness rather than things, is not a scientific position. It is a metaphysical position and a very problematic one that is running its course as philosophers and scientists try helplessly and in vain to explain things like consciousness and the objects of consciousness.