You could use the exact same argument for the swastika. The swastika became the symbol of nazism because the nazis decided it to be. It was a religious symbol for thousands of years at that point.
The key is to use context. A swastika on a temple that's older than nazism? Yeah, not a nazi symbol. A swastika on a flag at a right-wing protest in America in 2020? Probably not a religious symbol.