You are correct. I missed read that question.
If you select an easy passphrase for your key, it's similar to selecting a weak password for your bank account when your bank doesn't enforce password constraints. Buyer beware.
An ideal solution would be a two-factor key encryption. That would require the two-factor key to unlock the private key. There might be something out there like that. I haven't looked though.