Is there a way to prove that it is not caused by systemic racism? Having some unfalsifiable quality is pretty useless, and if it is the sort of thing that's never meant to end then that feels more like religion.
It's one thing if you produce a new model and present it at a conference, and one of the conclusion of your paper is that "this model does great on light-skinned faces but performs poorly on dark-skinned faces, we need to make improvements in optics and low-light sensing to bring up the accuracy on dark-skinned people." That's not racist, that's doing science.
It's another if you are a for-profit image recognition company, or if you otherwise deploy that model in production without so much as checking to see if your model generalized well to dark-skinned faces, or without attempting to add some post-hoc bias correction, etc.
We are currently in a world where the latter happens quite often. It's not just a race thing; there all kinds of inadequate "AI" systems being pushed on people that don't really work as advertised, to the detriment of some group or another (sometimes to everyone). But when it becomes a race thing, it's especially distressing to a lot of people, because Europe and the USA have long and grim histories of racism that extend up to and including the present day.
> ... if you otherwise deploy that model in production without so much as checking to see if your model generalized well to dark-skinned faces, or without attempting to add some post-hoc bias correction, etc
Neither of these would rule out systemic racism in either case, though, since, as you said, the problem is that "the outcome is biased against a certain group of people". Regardless of whether there were BIPOC people signing off or whether you had a perfect training set/etc, the only thing that matters is the outcome.
Mind you, I'm not a sociologist, so anything I say here is going to be a suboptimal paraphrasing.
Systemic racism is not a cause of something, but a description of a state of existence.
Systemic racism is itself the existence of inequitable outcomes in the context of a society that was literally founded on overt racism.
That something "is" systemic racism is falsifiable by demonstrating that one or both of those criteria is not met.
So this particular outcome will always be definitionally systemic racism until the outcome is no longer inequitable, or the USA has moved meaningfully past its white supremacist origins to the point where individual cases of accidental bias are truly individual cases and not part of a pattern.
Personally, I am okay with that. Ceteris non paribus sunt.