So what happens when someone's email account is compromised?
Currently there's a fair chance that the victim can reset passwords and change contact info for their online services before the hacker bothers to do so. But this would be impossible if email were used as the sole form of authentication.