Perhaps the other factor of 2FA is rather optimistically your yahoo username.

Somebody online would know my yahoo id, but only a microscopic fraction of online people would have physical access to steal my phone.

Someone who steals my phone would almost certainly have no idea what my yahoo login name is. I would have to look it up myself just to make sure...

The intersection would be close friends and family members, and if I can't trust them, then I'm totally screwed aside from mere yahoo groups login issues.