When I see people argue that systemic racism is a driver, they're not arguing that there's a conspiracy directly targeting people of color. Rather, they're arguing that racism drives things that create conditions where people of color disproportionately suffer harm.
A classic example of this from several decades ago is redlining, which prevented people of color from moving into safer neighborhoods, and undermined the neighborhoods they were able to buy homes in. Settlement patterns have long-term generational effects, so while it would be incorrect to say that predominately black neighborhoods have higher levels of asthma, violence, poverty etc as a deliberate plot, it's absolutely true to say that those are consequences of direct racism from several decades ago.
I don't think this should be a particularly controversial point. Where I think it gets more controversial is two points -- 1. What redress, if any, should people downstream from historical racism have? They're still suffering the effects, after all. 2. What patterns and decisions are happening _now_ that will similarly have downstream generational effects?
I'm not going to even attempt to suggest answers to those, just hoping to illuminate the framing here.