You'd have to look at your logs when it happens but google has a habit of tarpitting(i.e. repeatedly delaying email receipt) emails from servers they don't necessarily "trust". For large providers (i.e. those that would be used for handling one-time link emails), this normally isn't a problem however for small or self hosted email servers it can be from time to time.

I'm not sure what all goes into ~~the algorithm~~ that google uses to decide to tarpit emails from a server but I'd make sure that you have your SPF, DKIM, and DMARC all set up properly to give google as little of an excuse to dislike your server as possible.

It's not a great solution but I've long since given up on any attempt at self-hosting email and just use my own domain with protonmail at this point since the industry seems to be so hostile to self-hosting.