Before connecting each client needs to be set up with (1) its own private key and (2) the server's public key. The server also needs to have each client's public key. Once you have securely shared this information out-of-band, there cannot be a man-in-the-middle attack because both sides know the expected public key of the other side, and can prove ownership of their own public key.