True, in theory, though in practice, i know plenty of capable people but almost none of them bothers to read the openssh source (or even a subset, like recent changes) before updating or recompiling.
Basically, the story was that a program for grad research was inserting all kinds of nasty, anti-semetic things into text and it turned out the previous grad student had poisoned the compiler which was modifying the strings and was able to re-poison it every time through something else.
I forgot the exact details but it is an amazing read.
Then they aren't paranoid but normal folks, eh?