Does the client send the private key to the server for verification? I'm not sure but I doubt it. At least I hope that's not what it does.
My guess would be that the server sends an encrypted message of some sort that the client cannot decipher unless it possesses the private key. It deciphers the message then sends it back to the server in deciphered form to say: hey, see, I have the private key, here's the proof: this is the message you sent me earlier. I couldn't have deciphered it without having the private key.
Can this scheme be used with passwords as well? As in, treat the password as a private key, so the client never has to send it over the network.
Server ---- [garbled-message-challenge] ---> Client
Server <--- [ungarbled-message-response] --- Client
Does this scheme work?