But there are no guarantees about the source either unless I am willing to audit all of it?
* Being open source protects against a malicious developer. Otherwise there is nothing preventing him to build the binary with a different source, and send the passwords to his own server.
* Signed code archive prevents against a compromised hosting site.