Tools like bindiff have been around for years and take advantage of the fact that compilers don't randomize code generation. Instead, the callgraph and control-flow graphs largely reflect the structure of the original source code. Once you have leverage by exact-matching the parts of the binary that are nearly identical, you can build up and down the tree of nodes to find those that have more changes.
Crypto backdoors can be unbelievably subtle though. A single branch condition, a bit that is flipped, etc. can all lead to catastrophic failures. For example, a compiler optimization for dead code elimination led to some zeroization of key material being skipped. This kind of thing is extremely difficult to find and requires a careful understanding of the underlying code.
I agree with you that most differences can be found, but understanding the ramifications of those differences requires extremely careful analysis. A crypto flaw does not stand out from a mis-optimization.