In any larger environment or company, assertions like "everyone's keys have a strong passphrase" are basically impossible to make.

Multi-Factor into the VPN or SSH bastion is a good idea to look at.