Yes, it’s ultimately trust in the device maker, the compiler, the chip manufacturer, etc, and those can and have been verified to some extent.
But that’s not the point. A phone can listen or not, where you have no doubt with the cylinder. It’s its raison d'etre.