Probably not much of a risk if you're outdoors.

Personally I've found it to have an effect when indoors, even if there's an open window. (I don't live in an especially warm climate, though. Windows are not typically open all that far.) I had a friend who was a very heavy dope smoker and I'd often feel a bit stoned after leaving his house. Nothing major, just as if I'd had a few drags on a joint.

(I used to be a heavy smoker myself, years ago, so the sensation is quite familiar to me.)

I don't know why it's such a strange idea that this could happen. You are after all breathing in marijuana smoke. Second-hand or not, over a few hours it seems to add up.