Paying attention to diversity doesn't mean picking worse proposals. It means ensuring that minority groups in your community submit proposals in the first place. It means being aware that proposals written by minorities are likely to sound less confident and may understate the value of their content. It means ensuring that your conference has a clear statement of supporting members of minority groups who may feel uncomfortable in a space dominated by white men.

If your defence against claims of your conference lacking diversity is "Well, all our proposals came from white men", that's your fault. Countless other technical conferences around the world can attract proposals from community minorities. If you can't, you're doing it wrong.