It doesn't really matter if people didn't have a unique name for it -- I can distinguish tons of colors I don't have names for.
As you grow older, you learn learn the importance of color terms like "salmon" and "ecru". But you can be guaranteed you were perceiving them since you were little...
But claims like that are made as part of the everything-is-relative-and-culturally-determined movement. That denies men and women have any biologically determined average personality differences whatsoever, for example. Or that "harmful" emotions like anger are culturally imposed rather than innate, and all we need to do is "unlearn" them and maybe we'll get world peace.