I agree with you that we don't need an expansive surveillance infrastructure around drones (yet), but
> have not had a single incident involving injury or death
That is... very wrong. Don't do an image search on "drone injury" unless you have a strong stomach.
Drone cuts off tip of photographer's nose: https://www.brooklynpaper.com/drone-strike-our-photographer-...
Drone injures Australian triathlete: http://www.bbc.com/news/technology-26921504
Drone injures bystanders in Virginia crowd: http://www.washingtonpost.com/local/drone-crashes-into-virgi...
RC plane kills spectator at football game: https://www.nytimes.com/1979/12/15/archives/fan-hurt-by-mode...
RC helicopter kills 19-year-old: https://www.nytimes.com/2013/09/06/nyregion/remote-controlle...