We've had "magic" links as the primary sign in method for our site for about a year now. It works well but we very regularly get requests to allow password login. So much so that we'll introduce that eventually.
Seems like both in combo could be a nice solution, but I'm sure we'll just start getting support requests from people whose password doesn't work because they never set one...