So, to summarize, instead of trusting physical hardware owned by different people we are expected to trust that keys, that were used to sign genesis block will never leak?
Not quite - see other comments about needing a trusted local source the first time you sync up. Otherwise, forks are never tolerated. If the keys leak, and there's a new history, it will be ignored by all nodes who have been online.