You did the absolute right thing in that situation. I was in a meeting once with a CIO, middle management, and two female developers. At one point, the CIO got bored with the tech-talk and drove the conversation to his recently taking his son around to visit college campuses... and then onto the topic of which campuses had the hottest female students. The middle-managers immediately joined in the conversation, while the female techs stared at their hands uncomfortably until the meeting was over. The CIO was eventually dismissed for incompetence, but I will always regret not standing up to such incredibly unprofessional behavior.
In the 1990s, working for various government contractors, it was common for techs with military backgrounds to accuse male coworkers of behaving "like a woman" as a means of short-cutting discussions. As a nerd, I was often victim to those kinds of attacks. I'm incredibly thankful that increasing diversity in the workforce is creating a culture where those kinds of statements are no longer tolerated. Fostering a workplace that's good for women also fosters a workplace for men who don't fit traditional cultural norms of masculinity.