The big danger is from over-reaction and then just a law outright banning all civilian use. This is my worst case scenario.
When me and my friends fly it's a day trip. We travel to a remote location, or a large open flying field. We don't fly over people, or sporting events. We check local regulations over places banned from model flying. Where I am (Australia) there are areas where it is not allowed at all. All the inner city areas are off limits except for a few allocated fields. It's all handled at the local council level. Different councils are different.
We fly under the 400ft ceiling and give way to all other aircraft (Hear a real plane? Bring yours down.) We don't need any permits. I usually don't crash but I have in the past, and I know that I might. Components fail. Radio links can have issues. And then there's the wind and the elements. So we take all the due precautions.
I don't want people to ban my hobby because some fools take this new piece of tech and without much experience act recklessly with it. It's a danger with anything. There are young hoons who burn their tyres in the middle of the night on some corner. Don't ban driving. Or the car. Or all civilian uses of the car. Let's all keep our heads screwed on.