Whiteness and maleness are not and cannot be material facts and it is a category error to insist otherwise, nevermind elevating them as the axiomatic center of your worldview.
And more centrally, whiteness (in particular) and maleness are still not facts. You're in voodoo territory if you center them as such.
I fail to see the nit you're picking at with that second bit. Are you saying they are categories? Suppositions? Made up? Do you mean to imply then that they are "spooks" and their presence in the minds of physical agents has no bearing on physical outcomes?