> And of course, there are VPN tunnels between data centers in addition to the above.
Could you please be more specific on the VPN solution that you are using? How do you manage the shared keys? How do you make sure 'system administrators can't easily read the traffic?"
Which could be done legally via 'Cisco Service Independent Intercept (SII)' built into IOS to comply with CALEA (Communications Assistance for Law Enforcement Act). And not so legally via user-escalation exploits within the same service.
Anyway, props for making the effort. I too am interested in your key exchange methods.
Phone is I think an obvious (and now clearly wrong choice) although maybe always suspect if you are concerned with dark fibre . The endpoint security of a device generating and transmitting the key now also being a risk. How far up the chain do you worry?
An airgapped device to generate the key and a single person travelling between datacentres seems the secure (although costly) solution. Obviously if TSA/customs remove device from them for inspection or connect it to anything it needs to be thrown away (or moved to insecure duties) and the setup process restarted.