> But the key you are talking about the SSH client recognising is the server key right?

Correct

> which is why I said that if someone who is able to redirect your SSH connection to a different server

I will try to be more clear. Often when I am away I use ssh to connect to my desktop pc which has a dynamic IP address. Even though I have already stored the public key of my server in known_hosts it keeps asking me to verify if I trust its public key every time that it changes IP address without even mentioning that it has met said public key before. So if I am not careful enough I might end up accepting the public key regardless, even if I am being MITMed.