Generally through measurement. Colors are just differences in electromagnetic wave frequency, change the electromagnetic wave frequency enough and you are out of “visual light rays” and into other types of rays (radio, gamma, x, etc...). Until people start “seeing” those other non visable light rays, we have some indication we are all seeing the same things (light waves).
As to the “colors” of the various frequencies within the visual light wave spectrum, we may receive/interpret them differently (ie color blindness), but the wave frequencies are the same and it is us who work differently.
That’s why we can have other objective factors and measurements (primary colors, mixing of colors, etc...) that suggest for the most part we are all seeing the visual light spectrum waves similarly. In otherwords if you and I saw the visual light wave differently it should be obviously once we start mixing colors to produce new colors (if you saw black and white where I saw blue and yellow, then you couldn’t see green - it would be black). Still your use of “experience” of colors likely is inheriently true we probably all do experience the colors differently (some are my favorites, those don’t need to be your favorites, some may trigger certain emotions in me and not in you) but that doesn’t change the fact that the electromagnetic wave is the same for both observers.
But more importantly, I think it's like arguing whether all computers are big-endian, or whether some computers stores [R,G,B] values in memory as [G,B,R]. It's not actually mysterious.
If you point a measurement device at a particular point, you can get a histogram of light intensity over several frequency bands. A histogram with many narrow bands gives a more accurate profile of the measured light.
The human eye usually has only three frequency bands: blue, green, and red. They overlap. They are not always the same width, or centered on the same frequency. The width and center of the red bar is defined on the X chromosome, so some people with two X chromosomes from lineages that see red differently have a fourth histogram bar, and therefore have better ability to distinguish between similar color profiles at the red end of the visible spectrum. Some people with only one X chromosome have a defective copy of the gene encoding for the red histogram bar, and thus only have two bars in their histogram.
So while color is objectively the same, different eyes--connected to different visual cortexes--quantize and encode color information differently. Each brain may interpret the data in the color channels differently to establish an individual's world model.
What I see as "red" is unique to me. If someone were to bridge my brain to someone else's, my "red" would not match to someone else's "blue". My color channels would be as their octarine, smaudre, and refulgine--three completely different color channels that they have never experienced before. But if our optic nerves were linked, rather than somewhere deeper in the brain, I would see their red as my red, even if their eye reads reds slightly differently because of a minor difference in their X chromosomes. The translation layer between raw visual data and personal worldmodel is not guaranteed, or even likely, to produce similar results.
Because the words "red" and "blue" can be calibrated by reference to external objects. You and I both call stop signs "red" and the sky on a clear sunny day "blue". So if, for example, somebody invented a device that could "translate" your experiences into my brain, and the device "translated" your experience of looking at a stop sign into something that my brain decoded as "blue", I wouldn't conclude that your experience of colors was different from mine: I would conclude that the translator device was broken.
Sure, you can be taught that when you hear a deafening sound to say "wow that's really soft" but that doesn't change the fact that it's a loud sound and you're experiencing a loud sound.
You know that red is red, but you don't know that your red is anyone else's - which may be why I like green and you like red, we may have two totally different experiences of the colours.
However, I have rationalized away this theory with a thought experiment. Imagine someone messed with your brain and remixed your colors. So now maybe what you once saw as red you now see as green. However, since your brain has no other reference, you will not even be aware this change has happened.
Now if we put a device in your brain that changes your colors every second or so, you still won’t even know. You’ll never know! Because you have no way to compare a previous representation with a new one.
Imagine looking out now into the world around you, and being told every second your colors are being rewired. Since you can only have one dictionary of colors in your mind, you see nothing change, because that would imply there is an even deeper meta-representation of colors you can compare with.
Ultimately, colors do not exist. This is why my favorite color is black, as it is the only “true” color, and not to mention everything looks good in black.
So am I.
Some people lack nerve 'analyzers' and do not "feel" pain/heat/cold etc, just pressure: https://en.wikipedia.org/wiki/Congenital_insensitivity_to_pa...
Likewise some people lack emotional analyzers and do not "feel" emotions:
https://en.wikipedia.org/wiki/Antisocial_personality_disorde...
I don't think there's any way for me to tell which (if either!) is the more "correct" perception, eh?