Client must be open-source, and clients and servers must be interoperable.
But things like encryption help build trust. They ensure that there are safeguards in place in case something gets compromised. They reduce number of bad actors from unlimited to just one (the vendor).
In my opinion, all these are very good reasons to always opt for E2EE even if it isn't perfect.