Because of CYA syndrome, it's generally easier to pass new laws that to repeal them. If drones actually start causing the problems you suggest, it makes sense to restrict them in specific ways. But preventative legislation kills countless beneficial uses.
I'm flabbergasted that people worry about drones falling on them, then they go outside and think little of walking next to giant metal boxes zooming by. Can't they see how mistaken their priorities are?
I'm not going to comment on the 'invasion of privacy' part, as it's a strawman; there are no cameras that are easily available on these devices that can 'hover outside someones apartment' and spy on them any more reliably than a cell phone on a stick.
However, and this is a very important distinction; these are the small majority of all RC flights by all amateur pilots.
I agree that there should be stiff fines for those that break the regulations. I do not agree, however, that everyone should be subject to some crazy ill-advised extra regulations.