If all you do is download the release binary, anything could compromise you. If you read the source, an inauthentic release binary could compromise you. If you read the source and compile it, a compromised compiler could compromise you. If you read the compiler source, bootstrap compiling it from hardware code (reading all the source along the way), read the app source, and finally compile it using your compiled compiler, then compromised hardware could compromise you.

Every step along the way you increase the severity of the statement "If I'm fucked, then so is everyone using X". You stop when the set of people using X grows so large than you are sufficiently worthless in comparison.

you'd know for a fact the sources from which the executable you're using was compiled.

this would, in theory, allow you to make sure it doesn't contain any malicious code.

in reality, of course, this is rather impractical since you'd need to manually verify every single source file and every single line of code.