That sort of reasoning doesn't apply when you're talking about white people. You are talking about the only group that enjoys and benefits from a unique privilege they cannot opt out of, regardless of how they may feel about other races.