No that’s wrong. TOTP based 2FA is totally out of band with the only attack vectors being losing your physical device (or a device backup!) or leaking of the secret from the server side.
I have some of my more important services tied to an older Yubikey. The only attack vectors for it would be on the server-side and the physical key.