An analogy is gun control laws. Murder is a crime, whether it happens with a gun or without, so in theory there would be no need to regulate guns ("guns don't kill people etc."). But in most countries in the world we still regulate them.
Maybe it makes sense to regulate AI similarly? Requiring that some guardrails are built in to prevent it being used to generate fake information or information that helps to commit crimes. Making sure that it does not leak private information (and being able to prove this somehow). Regulating if and how it can be used as a therapist.
Although I wonder if it isn't too soon still for such regulation.
I am not advocating for this particular issue, I am pointing out the flaw in your metaphor -- you can't restrict the math but unless people figure out how to make their own wafer fab then you can at least try to restrict the ability to do it.
The concern is not that AI is harmful on its own, and that it will unleash a Skynet scenario. It's precisely that it will be abused by humans in ways we can't even predict yet. The same can be said about any technology, but AI is particularly concerning because of its unlimited potential, extremely fast pace of development, and humanity's lack of preparation on a political, legal and social level to deal with the consequences. So I think you're severely underestimating "some harm" there.
I believe a more realistic concern is that some workers will lose their jobs to automation as it becomes capable of running more and more complicated tasks, because it's cheaper to run or, more likely, contract to run some fancy software that can be taught its job in a natural language. And if this scales up it may require significant socioeconomic changes to work around the associated issues.