I don't know about the USA, but it seems that generally filming in public where you catch sight of passerbys and filming a specific person in public spaces are wildly different things.

If I stand outside your house and follow you all day recording your every moment in a public space, is that legal or has that turned into harassment? Certainly in many countries, filming an incidental person may be legal but filming them in detail and from every angle is not (eg upskirting)