The world's largest employer (US Government) uses Smartcards (and no passwords).
Quite possibly with SMS as the fallback option in case the soft token is lost.
We have a human in the loop as the fallback. It’s just a different attack vector. Brute force becomes a lot harder, but social engineering becomes easier.