[-You would be correct if-] It seems you are correct when the protocol works as described in the parent comment.

[-But it so happens that the-] Of course in OTP mode, the YubiKey protocol protects against replay attacks by using a counter on the YubiKey. This (authenticated) counter value is included in the messages that are exchanged during the authentication - and hence any replays can be detected/ignored as the counter value will be less than or equal to the last received counter value.

Edit (deletions marked with [- -]): I had no idea people used modes other than OTP with their YubiKey...