As much as I'd appreciate more open source for the sake of transparency, binaries provided on websites aren't guaranteed to match the source code provided and I'd assume most users are pulling binaries versus building themselves.
I don't necessarily have to trust each individual app on fdroid or in the Debian repos. I have trust the maintainers are building them properly, and those people are not the same people developing the core app.