The key might leak or be stolen, the server might be misconfigured, the valid user may still need not be root, etc.
Perhaps rethink how you view and solve the problem?
If you leave password enabled you need to harden the env as you be getting git by every script kiddy the second your public IP is ready.
You shouldn't be logging into these systems manually either - you should be provisioning them with the OS's default config tools like Ubiquiti or Cloud-init, storing the configs and creds in encrypted formats and the feeding them to chef/puppet/Ansible or any other configuration management tool.
Humans shouldn't be fucking with production systems, they should be making commits to git that are applied by automated systems.