Yes, I agree!
But what if my ancestors did not steal your ancestors stuff? Am I still responsible, because I have the same skin color as the folks who harmed you?
This is fundamentally a race-based conception of the world, a conception that defines power along racial lines, which groups and analyzes people fundamentally based on race.
We have the benefit in the last 50 years to move past this way of looking at the world, many of us have come to the realization that race isn't a useful way to group people, period. Any disadvantage you see in society that appears 'race' based can be better explained via other means. If people aren't treated justly, it's not because of racism except in vanishingly small amounts, in obscure and backwards parts of the US.
'Structural racism' falls prey to correlation is not causation, a misguided explanation for group differences, an oversimplification, and it won't generate progress as long as the cause is incorrectly ascribed.
Here’s a misunderstood concept: race is a result of unfair policies. “Black” as a racial group does not meaningfully exist without the historical racism and oppressive policies enacted on black peoples. Without distinction between groups, there would be unity. But we know, for example, black people were treated unfairly and still are in many aspects today in the US, so the construction of the black race occurs because of these differences. Aka, the marker “black” for a person identifies someone who faces systemic racism in the US. It’s not just about the wavelength of light a person’s body reflects. Racial groups are the result of history, culture, policies, and present day attitudes.