Or just set `IdentityFile` with the path to the key and set `IdentitiesOnly` to yes for the specific host you connect to, and disable public key authentication by default for everything else.