My Raspberry Pi some time ago had a setup where only public key auth was enabled for LUKS unlock, so I only had to have an authorized_keys file unencrypted.
Correct, someone with physical access could run a MitM attack and steal your passphrase. I just find this extremely unlikely, so I honestly don't care.