It has only happened to me once or twice, and is a non issue if you choose which window(s) to open carefully.
That being said, we love our total home fan, and it's also great for clearing out various odors
Edit: if people are having campfires near your house you'll want to shut it off asap