@johnsusser is exactly right. You can't add diversity at the end.

So if you want to have more diverse people speaking at conferences, make sure you get more diverse people interested in school. Don't give a conference flak because biases in school and society have emptied their prospective diverse speakers to a pool of zero people.