What should or shouldn't matter is irrelevant and pretending like it's relevant is naive. It does matter. That's a fact. You want to take your argument to 1850 and argue what should and shouldn't matter? We fought the bloodiest war in American history over this so you know what, it does fucking matter. And even in 2020, racism is still widespread, so you know what, it fucking matters. That's reality. It most certainly fucking matters that certain people were held back for hundreds of years through slavery and racism while whites progressed forward. It most certainly matters that this is still going on today. Look at the median household worth in the US to see how much it matters. This idea that you're putting forward, that history doesn't matter, that reality doesn't matter, is frankly absurd. It's not even an argument. It has no basis in reality. It's a fantasy. History matters and affects the present. It's as simple as that.