Yes, that is a fair assessment. Unfortunately, I don't know what the solution to this is.
The solution is mentioned above. Unironic nazis/white nationalists/white supremacists will only last long on free-speech-oriented instances which are likely to be defederated from any instances which explicitly disallow hate speech. Pick one of those latter instances as your home, and you will most likely never see a "nazi" on your timeline unless you go out of your way to go hunting for one.