You realise conference organisers can do the very same thing without a code of conduct, right? That a code of conduct can be used as an "oppression tool" doesn't mean that its absence doesn't lead to even more "oppression". If you want to compare the relative merit of two things, like having a code or not, you need to consider both sides rather than just point out the perceived downsides of one.