It's worth noting that they specifically state that for Gmail they can use OAuth through the refresh/access token system. Hopefully they do this for all other IMAP servers that support OAuth (there are a couple of them now).
However, for traditional IMAP/SMTP servers your point is well taken. They must store your credentials in a restorable state, which, however carefully you do, has difficult security implications.