Well, the sysadmin could click on "restore password" on that super-secret site they want to steal your password to, and then check your mailbox. So this is really no different in terms of security.
Also, I'm sure I have a dozen restore password emails in GMail somewhere. And I'm sure I'm not alone.