Fair enough. I had the same complaint about regular smokers at my last apartment. Smoke from a neighbor would somehow seep into my place a bit, which bugged me. The upstairs neighbor had kids and a dog that irked me with their constant noise.
I solved the problem by buying a house.
Freedom has it's perks and problems for all involved.