This is a disengenuous argument. Even people who genuinely like diversity are not obligated to want to live near all possible kinds of people. I like diversity, but I don't want to live next to neo-Nazis, anti-vaxxers, serial killers, etc.
The idea that liking diversity means liking all possible humans is flawed.