Sure, Japan is a ~monoracial country and there isn't the level of racial awareness there that you have in the US. I feel like you have to accept some of this if you choose to live in Japan as a person of color.

I experienced this myself when I wanted to do the "teach English abroad for a year" thing in Japan after college. I have Top 1% English skills and have won literary prizes for my writing in college, and speak in a normal American accent that you might find in a news anchor. But everyone wanted a white person.