When say you chose people solely on relevance and quality, but you only have men as speakers.. Then it's laughable that your system for choosing speakers isn't biased.

Percentages shouldn't be the reason why you hire or don't hire someone but they should be a measure of how much bias is in your system for choosing persons of varying classes. If your percent of xyz participants in the majority class is way higher than the distribution in the possible applicants then there is some sort of class bias(economic, race, socio) in either how you choose applicants, or in how you get applicants.