Sometimes this is a necessity. In history class we learn about ancient Sumerians without inviting them to the discussion, because they have all been dead for a long time. So we make do.
Certainly, if we had the chance, we'd love to have them there.
(A useful thought experiment: what if a bunch of ancient Sumerians fell out of a time machine and were available for discussion? Would we not think poorly of people who wanted to discuss ancient Sumerians did so without the Sumerians, who were standing like... right outside the room?)
Can a bunch of e.g. men sit around and have a meaningful discussion about women? It... really depends. Certainly a group of male gynecologists could usefully discuss some procedure, although the history of gynecology also certainly suggests that for many years the profession surely suffered from a lack of female perspective. Men can usefully discuss gender relations, because they are a part of the group that experiences gender relations, although again this discussion will be a lot weaker if it involves exclusively male perspectives. A room full of men can probably not very usefully discuss issues highly specific to women, unless they were indirectly drawing upon female experiences (e.g., two men could certainly learn from a discussion about an article written by a female author about childbirth or male-on-female sexual assault, etc)