I think the problem lies in defining who a Nazi is. At Birkenau it is clear, and for a lot/most/all of those chanting ‘blood and soil’ in Charlottesville it is pretty clear, too.
There are many shades of grey between right wing and Nazi, though, and if we decide that Nazi speech is to be banned then we also need to start drawing a very clear line as to what is ‘Nazi enough to ban’ and what is just ‘approaching Nazi’.
I’m live and work in a country where the public denial of crimes against humanity is punished by fines and prison. I don’t deny crimes like the Holocaust or Srebrenica or Rwanda and I despise the people who do, but I still feel uncomfortable with laws that criminalise that sort of speech. This situation is not exactly the same - Daily Stormer can probably find someone else to support their site - but there’s something to the slippery slope argument.