I find it notable that you threw in panhandling and "other nuisances" right up there with doing drugs in public. They don't seem comparable.
There's value in being discerning about one's fears, and it's probably best to get a handle on the less justifiable ones before sending the state after people.