Server infrastructure is not Open Source (I guess). So, this might be a problem for some.
Exactly...it is a problem for me.
If the proper application of E2EE can be validated using the open source client apps, why is the server code relevant from a security standpoint?
Host your own server, build your own client, and yet be able to talk with everyone else.