The racists on the left think that we need to pump minorities full of welfare and government services in order to better subjugate them for their own benefit. The non-racists on the left think welfare and whatnot will help those people.
The racists on the right think that the minorities are a lost cause, dead weight to society and that we shouldn't waste a penny on them. The non-racists on the right think that we we got the government out of their business as much as possible these people would be able to help themselves.
Whichever group of non-racists someone's world view aligns with you will have a hard time spotting them on your own side because their racism just looks like good policy and you will see many more on the other side because their bad policy looks like racism.
The republicans probably do have more racists in their base because they're the whiter party and for historical "convince the poor white man he's better than the black man" reasons but IMO when it comes to actually pushing policy the racist party is whichever one has more policy you think is bad.