But that's the rub though: How do you define that there are groups in the first place? There is no way to land at our socially-defined set of races via objective means.
"Yellow" and "Orange" are words, not facts about the universe. They correspond (loosely) to physical observations, at least when it comes to perceived colors (there are plenty of optical illusions which reveal the vast gulf between 'the color a thing is' and 'the color we perceive'). So I don't really know what you mean by "real" here, but is it an objective fact? No.
That's a philosophical position called nominalism. You can be a nominalist but other people disagree.