Actually modern society protects those that have colour vision defects much more than societies of the past. In the past if your vision wasn't perfect you tended to die, now days you can be completely blind and live a long life.
Colour vision is not much help under low light conditions which is the reason our distant shrew-like ancestors lost their colour vision in the first place.