If the Nazis had used a common letter like Z, there would not be the same stigma attached to the symbol. That this is so obvious suggests your argument has a flaw.
I’m curious how one calculates that.
Also: don’t mistake an observation for an argument. I’m not implying anything. I think it is an interesting thought.
The swastika is a bit more complicated because the western world's exposure to it isn't all that big, but I don't see how a symbol that is used by probably over 10% of the world population today and has been used for over 5000 years should be shunned, especially when you consider that it's not even the same symbol most of the time!
At the end of the day most of this stigma wouldn't be an issue if people would just stop assuming malice from others by default.
[0]: https://www.cbsnews.com/news/ok-symbol-hand-gesture-anti-def...