It affects communication. People can still discern the difference between colors, they just don't have an easy way to communicate this difference to others.
The Japanese language until relatively recently didn't have a clear verbal distinction between what we call green and blue in English. That doesn't mean Japanese people can't tell the difference between green and blue. It just means that there is a kind of "blue" that is the sky and a kind of "blue" that is for traffic control lights, and in context nobody is confused.
The same issue can occur within a language between people with differing levels of study of color. A graphic designer might say a particular shade of green is "chartreuse" that his boss instead might call "yellowish green".