I don't know your specific situation, but most people use password-based sites that have an "I forgot my password" feature that authenticates based on email. Thus most people have a single point of failure already: their email account.
You're right about email being central to authentication on the Web. This makes it important to protect.