The other area familiarity with whiteys isn't exactly a positive is in some towns around Hokkaido. There, the experience is with russian fishermen, and the sentiment is clearly visible on signs by the door: "no foreigners".
All that said, I never was that bothered by the mild xenophobia one experiences in Japan (and elsewhere in east asia). It's bemusing more than anything else. I would imagine that non-caucasians have it somewhat worse, though.
https://en.m.wikipedia.org/wiki/Yokosuka,_Kanagawa
Edit: the article also fails to mention the huge stigma attached with "abandoned" or bad luck homes .. If there's even a hint that someone died there, or that the abandonment was the result of hard luck it can damage a property's worth. Not to mention earthquake readiness of the homes - something on people's minds a lot since Tohoku 3/11.