If you find this interesting, I recommend checking out the Radio Lab episode on "Colors". They go as far as speculating that ancient cultures perceived fewer colors than we do, because colors like "blue" aren't common in the natural world so we never learned to "tune into" the visual stimuli that tell us something is blue until we started to create more blue things.
Everyone isn't hearing the same word.
There is a significant difference between people classifying things differently when they exist right on the boundary between two classifications (ie two implementations of the same classification method will frequently disagree), and classifying things differently when the classifications are nowhere near each other (ie you’d hope an objective method has a significantly lower error rate).
The former is about the accuracy and precision of your measurements. The latter is about how you interpret them. Both are important, but are also different subjects.