If they had proper reproducible builds it wouldn't be an issue… Also, why do you think fdroid is more likely to backdoor it than they are themselves?

With fdroid i trust 1 party, by downloading random binaries I have to trust hundreds.