Scenarios where someone might have my password but not my SMS: keyloggers, password written down at my desk, video surveillance or shoulder-surfing, audio analysis of keyboard and/or CPU tones[0].
The stronger or less guessable a password is, the more likely it is to be written down or recorded somewhere where it could be compromised.