I don't know anyone that has spoken to a former slave.
And most people of color in the US's ancestors were not slaves in the US south. They came as immigrants from other countries.
The issue is not slavery. It might be racism. But it is not "I can't have slaves anymore"
That certainly means it's recent to me. My own parents (both still alive) grew up while "separate but equal" was still law of the land.
So, yes, it's definitely racism. Or shall we say, the lasting imprint of slavery and the Confederate ideology more generally. Reconstruction failed and therefore allowed the Confederate ideology to persist for 100 years of Jim Crow and right on into today.