The idea goes something like this. You have white privilege because you are visibly white. By being visibly white you gain an automatic differentiation from those who are non-white. The belief is that the system, being largely run by white people will extend courtesy and nicer treatment to those who look like it whilst those who differ in appearance must "earn" the position you are given by the elements of your birth. It's a bit more complicated than that but that's my understanding of the core of it.
I am not trying to strawman the argument and if I have it wrong, please explain where I wandered off course.