It's easier to trust that the maintainer compiled upstream properly instead of backported/mismerged because of a stupid OS policy preventing you from incrementing version numbers
In the end you will always arrive at a chicken and egg situation, you will ultimately need to trust the engineers who designed your CPU and chipset, the VLSI design software which they used, the developers who wrote the compiler and toolchain, the tools used to bootstrap it, external libraries, etc.
The world ultimately runs on trust, no matter how you slice it.