Not speaking up when such a comment is made is doing something wrong.
Not to diminish the fact that even raising it to HR could be challenging in some orgs (e.g. see Susan Fowler's saga at Uber).
And also worth noting that it's easier to find the confidence to report to HR in an org where the line has been clearly drawn by actions such as the dismissals that we're discussing -- that's the point of the "shocking rule" principle.