If you disagree that potential is a good enough justification, would you agree that private citizens should be able to own nuclear bombs? If not, why? If you do think citizens owning nuclear bombs is reasonable, I don't think we will find common ground.
> Anyone can carry a poison or a pathogen or simply be highly trained and know a lot, no need to carry anything, or even be obviously muscular.
Try attending a conference openly carrying a bucket of arsenic or a bottle of anthrax. Or just a jerrycan with gasoline and some matches. Or wearing a bomb vest and wielding a machete. I like to imagine you will be sent away. It's not like guns are unfairly targeted here.
When those other methods of murdering people are as common in the death-statistics as firearms (such as the "killing spree by a muscular martial artist" one you propose), perhaps we should worry about them more. But they are not, so banning guns is the most logical step to increase safety.