But it could be mitigated by "click the link and enter the one time code we gave you at sign-up time". Too much friction? How about "click the link on the same browser you used to sign up, and we'll verify that using a cookie we just set" - functionally equivalent and probably works for 90% of users while the rest can fall back to the one time code.
I've seen a handful of sites do something like this in practice. No idea why it's not more common: presumably most people don't roll their own verification process so if some major web frameworks adopt it we'll eventually see it more widely.