Er, no. IMAP requires you to send password (encrypted or no) every time you connect.
* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH
250-AUTH LOGIN PLAIN XOAUTHMy point still stands though, which is that you can use IMAP without passwords.