These devices will also become the ones responsible for setting up the same login on other devices. These would work in a way similar to, for example, how Apple or Okta handle two factor authentication by sending a push notification to existing devices registered to the account.
Since most people wouldn’t have a Yubikey (or multiple Yubikeys for backups) and since most people own one smartphone, the fallback for account recovery — when their smartphone breaks or is lost — would be the weakest link.