It can't be some property of them, because believing that would be racist, right? If it's not them and it's not us (I'm white) then what is it?
But I'll bite anyway: historical racism clearly still plays a significant role. Whatever the level of present-day racism, I'm sure we can agree that it's orders of magnitude lower than it was historical. The plight of people of colour today can be attributed to historical widespread institutional racism independently of whether any such racism exists today. Also, your model fails to account for the possibility that racism can exist and be harmful without being widespread, and that people of colour can make bad decisions without that being a "property" of their colour (shaming people for "acting white" strikes me as particular counterproductive).
That's not saying that's the whole explanation (or even that it is the explanation), just that your model ("it's not them and it's not us") is a false dichotomy and generally too limited to be likely to yield a satisfactory answer.
I think your argument is sound. Axiom: there's actually not that much racism. Theorem: the remaining racism is confined to a small number of people.
I just disagree with your axiom.
Why does it need a different explanation then why many people of the white persuasion keep having a bad time? When you selectively group people together, you will always get an identifiable pattern. But you can't then claim that the pattern is universal, since the pattern was predefined in your selection.
I also collect explanations for why there is so much violence against white people. I think a lot about violence against men lately. But that doesn't stop me from thinking there are some instances of violence that are best explained as violence against women.