As a user, I don’t prefer it, but I’m also very comfortable using password manager.
We implemented it at Doximity but I’m not sure how it changed sign in experience/metrics. It’s gated behind the “forgot password” link now rather than the default/only option. https://auth.doximity.com/magic_sign_in
For example, user@example.com signs-in to your site and stores data they expect to be private, they stop using the mail provider and the mail provider deletes the user's account which allows it to be re-used, then another unrelated user signs-in to the site and takes-over the account.
I imagine the only viable way would be a whitelist of domains of mail providers that are known to not recycle email addresses (like Gmail), or to also check if the WHOIS data for a domain changed.
Any service that allows resetting passwords by email has this risk, unless the service supports MFA and the user configures it.