The only remaining legal way to separate is via money. In other words: fuck the poor, they are not allowed to decide with whom they associate.

Instead of trying to fix the problem by allowing non-monetary ways to segregate, which would be available to everyone, people are busy trying to universalize the problem.

To answer your question: you're not an asshole, but you're the only one without a broken leg... and we'll break it to make things fair to everyone.