To get back into your account you're provided backup codes that you're supposed to store somewhere safe. Otherwise, if your phone were stolen you'd be out of luck, yes.