Agreed. In matrix/synapse, even with e2e enabled the server gets a ton of metadata, including presence info. Also the identity server exposes quite a bit.