TLS is already an under-spoken requisite of SMTP communications from any webserver doing "Forgot Your Password" workflows. TLS between mail servers is also already a de facto SMTP requirement for most bare minimum spam fighting mechanisms.
Let's not forget that most of the email system is still not encrypted (last time I checked anyway) even though the last mile between your mail server and your client may be.
There are also already a growing number of users that use "Forgot Your Password"-only login (most accidentally; human memory is porous) day to day.
Which is not to say that we shouldn't make email better and more secure, but that the security of our email infrastructure is a red herring of sorts in the larger security discussions of "passwordless" login, given that we are already equally insecure in every website that has a "Forgot Your Password" button that accepts anyone's email address.