I see drones as different than, say, smoking or skateboarding. Smoking irritates people, but outdoors, it is really easy to get away from. Skateboarding is potentially dangerous to bystanders, but it is very obvious when someone on a board is getting close enough to cause damage.
Drones can (and occasionally do) fall out of the sky. Knowing there's one somewhere over me leads me to pay a ton of attention to it, which is (obviously) extremely distracting. This is a case where it making less noise is actually more stressful, because you have to look.
It is a different degree of cognitive load and ambient stress compared to a skateboarder. As an example, I was trying to read in a park some time back, and found I couldn't pay sufficient attention to the book because of the drones. So I left.