Now realistically, most people can do that, so they have to trust that somebody who can actually did it. Security experts have been and are looking into messengers like Signal: they have an interest to do so as security researchers, because that would look good on their CV.
Of course there is trust somewhere, you cannot do without trust. But Signal is amongst the best you can find.
See my other question. Audit how? Sure, you can examine the source.
But how can one tell the app blob in their phone is the same as the one produced if you build the code?
https://github.com/signalapp/Signal-Android/blob/main/reprod...
(I haven't tested this myself.)
Edit: You can also build the client yourself and use your own version. No need to use the Google Play store version.
The problem being that apps as installed from the App Store come (to my knowledge, could be wrong) encrypted. So you can't just compare an unencrypted local build with the encrypted installed app (and you can't decrypt the app or encrypt your build the same way, because you don't have the key, e.g. Apple has it - and iirc, there's no access to it, because it's held in the secure enclave in iOS case).