It's very simple, the vast majority of popular chat applications out there don't have open source backends. Which means we basically have to trust their owners to do the right things and hope for the best.
The business model of choice in this space is to (sometimes) have partially open source clients but almost never use OSS in the backend. There's a lot of secret sauce and magic that these companies use to differentiate from each other and most of that is proprietary. This provides these companies with a control point to lock in their users to their network.
For the same reason attempts at standardization of protocols and federating chats and calls between networks have largely stalled/failed. XMPP is technically still around; it's just that none of the popular solutions in this space use it.
I'd say Signal is the positive exception in this space where both server and client are OSS and the client UX is pretty decent. Telegram talks a lot about security as well but their OSS seems limited to mostly client side. Of course a lot of the crypto is client side so depending on how paranoid you are that may or may not be good enough. Either way there probably is a lot of server side stuff that is relevant to end to end security that is not being scrutinized outside Telegram.
For both, even if the client is OSS, what goes into the app stores may still include stuff not accounted for in the source code. Auditing the source code and the binaries are two things. And then there's the runtime environment to consider, which is a OS that is probably proprietary that includes lots of stuff that is a bit icky from a security point of view.
So, there's multiple levels of "trust us, we know what we are doing" that you'd have to buy into in order to feel secure. IMHO that has been a problem for a while but most people/companies seem to be indifferent when it comes to security and happily pay through their nose for 100% proprietary security snake oil peddled by the sales people of e.g. MS, Zoom, Slack, etc.