I had that exact same conversation with a Japanese American who was raised in the Bay Area but moved to Tokyo in his 30s. “You get the gaijin pass,” he told me, “I don’t.”
They love Japan and have kept their Japanese citizenship, but all the rules and expectations were overbearing, so for them there are a lot of pros and cons to the two countries. And they call out that for foreigners, they get to see many of the pros but fewer of the cons.