This is assuming white privilege is the same as white supremacy, when the term white supremacy has been used for KKK and neo-nazis groups, not mainstream white society since after the civil rights era.
It also assumes that most whites and only whites benefit from white privilege, otherwise it's not so white, and may be more a combination of class, culture and/or historical consequences. Also the fact that white people are still a majority in countries like the US, where a majority in any country likely has similar privileges just by being the majority. One last assumption (in America) is that white culture is a certain way, when in reality the US is primarily an English dominated culture historically, whereas Europe has a lot of cultural variation.
A related issue is that white supremacy is sometimes extended to considering an entire economic system as racist, just because history went a certain way. But there's nothing about an economic system that says any one particular group need benefit more than another.