It would be fine if the users opted into this, but it happens silently and arbitrarily by admins, oftentimes based on speculation or gossip, not even real abuse. It’s all of the worst of tribalism, polarization, guilt by association, and preemptive censorship (regardless of whether or not admins are “in the right” by censoring messages between x and y flowing through their own machines).
It’s also not easy or reliable to migrate your account between instances without losing your followers, and none of the server implementations yet support virtual hosting, so you can’t migrate hosts while keeping your own domain/handle.
There are real problems there, and casually dismissing the major censorship issues in the ecosystem doesn’t begin to solve any of them.