As with many things, it depends. If you were planning on enabling email-based password reset anyway - as most sites do - then there is no degradation in security.
Magic link tokens are one-time-use anyway, so assuming you follow the authorization code spec, if any attempt is made to use the code a second time, you should revoke any existing auth tokens which resulted from that code. So the risk here is pretty low. You can also further mitigate this by implementing PKCE, which prevents any other client/app/whatever from using the code.
Also bear in mind most apps are able to store access/refresh tokens securely, so don't tend to have any kind of auth timeout. For most apps which allow magic-link login, your "session" will never expire, so it isn't particularly problematic to have to log in via an email link.
For a website, there's no major reason you shouldn't have a reasonably long-lasting cookie to prevent any need for regular login.
If you're concerned about long-lasting sessions, it might be appropriate to use a second authentication factor for sensitive actions - e.g. purchases, transferring money, etc.
Other than pairing with MFA/2FA, there isn't really anything meaningfully more secure than (properly implemented) email magic link auth. Assuming you want decent usability and your audience isn't overly technical, magic links are definitely the way to go as a primary authentication mechanism.