Something that helped me understand why this problem is significant: the things you mention as giving an account of color perception are only accounting for a brain system which evolves through states having structure which is isomorphic to fluctuations of colors in subjective experience[0]. However, the philosophical question isn't about whether you can create such a mapping, it's about reconciling the fact that if you look around your environment it's like something, and what it's like is importantly not a bunch of neurons. And yet when we look inside brains, we only find neurons. So if we want to say that the experience of colors can be explained physically as well, then there should be some physical place where e.g. redness is occurring rather than electro-chemical flux of neuron interaction.

[0] Another subtlety here is that the brain states having structure isomorphic to something in subjectivity doesn't necessarily make sense. It would on the assumption that everything is 'structural', but that's not an assumption everyone is willing to make. (You can find demonstrations that mathematics and the sciences can only deal in terms of structure, which is why for their practitioners it makes sense to ignore anything non-structural—but this is very different from a demonstration that everything actually is structural.) In that case, in order to create this mapping, we have to create another structure (for instance, a taxonomy of colors names and relationships between colors), and we can configure that structure in a way which is intended to reflect some moments of subjective experience. Now that structure could be potentially be found isomorphic to some evolving brain states, while to say that the subjective experience itself was isomorphic is likely nonsensical.