I would really prefer to be at a diverse conference. To make the conference experience as good as possible, maybe it would be wise to actively seek out diverse speakers if they are not applying.
FOSDEM is a diverse conference, it's great. If you've never been, go and then criticise if you don't like it. Don't rely on second hand information to make up your mind.
No, it wouldn't be. If those "diverse people" want to be at FOSDEM, they will need to follow the same rules everybody follows.