It is really worth considering that this conference used a blind review process[1]. Subsequently, there is zero chance that the reviewer's decisions were influenced by racism or sexism.
As a society, we need to consider the implications of what is going on here. In what way could this move possibly be said to be anti-racist/anti-sexist? It just isn't. If anything, abandoning your speakers because they are male/white (presumably), is itself bigoted. Consider how it would feel to be bumped from a speaker line up because of your race/gender.