As long as the email account is secure, and the throw-away one-time passwords are good, you have the frequent-rotation passwords security advocates dream about. Indeed, hand them a secure password they have to use (and forget).
I just made it an option in the "I don't have a password" form that instead of setting a password it just logs them in. So they don't even see / have a password to remember.