Consider this paragraph from the Signal protocol
> ...For example, they may compare public key fingerprints manually, or by scanning a QR code. Methods for doing this are outside the scope of this document.
> If authentication is not performed, the parties receive no cryptographic guarantee as to who they are communicating with.
I have not known anybody that actually does this. If an active attack does happen on somebody, I'm pretty sure they would not notice.
As far as I understand, a decentralized model solves this problem. Don't trust the server? Run your own instance or have some other way of validating clients (via your own PKI infra or similar).