It's not what or whether they were watching, it's how they were watching. It's a judgement call but pretty easy to tell if a group is leering or just watching friends have fun.
And that is why the "making a woman feel uncomfortable constitutes sexual harassment" standard is problematic. And of course it's not entirely wrong either because it certainly can be done on purpose.