If you want to make the panel more inclusive, you don't yank someone out - you make the panel bigger. Add more diversity and the session can get better. (Granted, this only works up to a certain number of panelists - I've been on a 20-person panel, which was a joke.)
The whole point of panel discussions is to get multiple viewpoints on issues of opinion. If you want purely subjective technical answers, that's what single-presenter-sessions are for. Panels are for the merger of people, process, and technology, and for that, diversity rocks.
There was a study performed on young students, concerning their perception of STEM fields. Most respondents indicated that "techie" people are all white males in lab coats, and that the respondent wouldn't fit in well with that group. (Sorry, I don't have a link to the study. But I know I saw it in the last 8-12 months)
If more women are publicly visible in tech fields, then young people's perception of techie people will swing away from "homogenous white male group" towards a heterogeneous group that is necessarily less discriminatory. This should cause a corresponding increase in enrollment by non-white non-males who would otherwise have been dissuaded from joining.
If everyone waits for the rest of society to change, it never will. Note that I'm not saying any one proposed solution is the right one, but it is an important discussion.
Just because you have good intentions does not mean that any random idea you put forth will achieve those good intentions.