>Sadly, having an open and explicit CoC and governance structure gives these people a weapon to use against you.

Also, not having one also makes lots of people rally against you. You can't win on the CoC front