This is a fair point, although I have many Google Accounts verified with my single personal number and that doesn't seem to be a problem for them.

I don't like handling passwords and it complicates the signup/login process which is why I don't usually implement password auth for small projects, but in this case I should probably consider it.