Red being green makes no sense. Let's simplify a bit and say you have a neuron for hearing the word "red", one for the visual stimuli "red" and then a sensory fusion/abstract concept of red. Several layers up you'll also have one for a "stop sign" (which also takes inputs like octagon shapes and some white letters). There are many other, more distant associations of course, we have many neurons after all.
The spoken words, physical wavelengths and stop sign design are more or less physically or socially fixed. If you magically swapped some red and green neurons then those physical stimuli wouldn't change. The brain would be forced to rewire all the associations until it arrived at the original connections again. Or it would suffer from observable mispredictions such as being more likely to respond to a green rather than a red stop sign. Why would there be a free-floating "qualia green" neuron connected to "visual red" that is yet somehow disconnected from "physical green" and why wouldn't it update once it became miswired?