It's crazy to me that your worldview is so deeply embedded with tribalism that you think your dad being a mechanic or your friend being black have anything to do with who you are.
The point about minorities is just that: by arguing that you're not racist because minorities are in your tribe, you're showing that you completely miss the point. For you, the in-group is your frat, your world. And being outside your world means lacking a variety of privileges in life.
But race is an in-group too, and maybe that's not one of the tribes you recognize, but for many people it is, and it's the tribalism that's the problem, not the fact that skin color happens to be a particularly easy and dangerous arbitrary criterion for admission into the tribe.