I've experienced that too.
This usually happens because whatever service the site is using to handle sending out emails is under heavy load, having issues, or is rate limiting the site due to a sudden spike in logins.
If this happens for any reason, it failure cascades because the more people try to log in, the longer it will take for everyone to get a valid link.
At least with a standard TOTP token or the like, if the service is under load it will eventually work if you keep trying (versus not working for longer the more people try).
Hell this method also runs into the issue of getting flagged as spam or getting blacklisted due to ~~the algorithm~~ for one reason or another. This is doubly likely if a sudden burst of traffic to a site results in them sending out a mass wave of near identical emails.
Now non-tech users can't log in at all since they "never get the email at all".
Email is a wonderful thing but it is painfully fragile and doesn't handle time sensitive stuff well at scale (due to queuing, routing issues, and spam filters).
As an email layman, I don’t fully buy this, can you explain? I’ve sent emails with multiple CC’s and most get it quick while one or two people have to wait for it to hit their inbox for multiple minutes. I assume they all leave my service at same time and the delay is caused by the receiving mail server.
The issue can occasionally be the user's email provider but for the most part issues with implementing one-time links will be the site's responsibility/a problem on their end.
As for email occasionally being super slow, Google and Microsoft (to a lesser extent) will tarpit(repeatedly delay the acceptance of emails from a specific address or entire domain) emails arbitrarily. This normally isn't anything meaningfully repeatable and is essentially ~~the algorithm~~ arbitrarily deciding it dislikes some certain email, email address, or server. Those emails will eventually get to their recipient but effectively get frozen in time for a bit before getting delivered.
For normal person to person emails, this boils down to bad luck and occasional inconvenience however for any heavily templated email (like say an email containing one-time links), this means something on the site's side is causing the emails to appear as spam or spam-like to the recipient server. So technically it is the user's email server causing the delay but it's usually due to something in particular on the site's side be it a domain or server configuration or something about the email contents.
On business systems, sometimes DLP rules impact delivery times.
That is not completely true. I've been operating my own mail server for almost a decade, since I am the only person using it the volume of email I send out is almost none.
I've observed on more then one occasion that sending an email to my gmail or my fathers gmail account can sometimes take 20 minutes or more to show up in gmail inbox although I know that google accepted the email as soon as it was sent.
I'm not sure what all goes into ~~the algorithm~~ that google uses to decide to tarpit emails from a server but I'd make sure that you have your SPF, DKIM, and DMARC all set up properly to give google as little of an excuse to dislike your server as possible.
It's not a great solution but I've long since given up on any attempt at self-hosting email and just use my own domain with protonmail at this point since the industry seems to be so hostile to self-hosting.
I do have DKIM, strict SPF and DMARC policies, MTA-STS, SMTP TLS Reporting, latest TLS support and valid certificates, there is nothing else I can do on my end.
Actually, it could be either. Postmark (the email provider for a bunch of sites) monitors "time to inbox" for several user email services: https://status.postmarkapp.com/. Apple, in particular, seems to frequently add 2+ minute delays on the receiving end. (The "source" link above the charts describes how Postmark collects the data.)
I mean, when you send batches of mails to gmail, it lets them through first, then depending on its whim it starts throttling or outright blocking them.
Once in a while, it changes its handling of DKIM or SPF or god knows what and the only thing you can expect from them is an SMTP error message.