If they're not threatening then there's no point in punching them, beyond perhaps satisfying one's own violent urges.