I lived in Japan and know many people in this situation. Most people just never tell the government and keep both. There's no proactive outreach by the government for these cases and it's ok unless you happen to mess up and produce your foreign passport when entering Japan instead of your Japanese passport. It's a "don't ask, don't tell situation".

There was a lawyer, originally from Japan, living in the US who did try to force clarity on this issue and sued the Japanese government. Unfortunately she lost so the status quo remains. https://www.asahi.com/ajw/articles/15077494