It doesn’t scare me because in Authy you also set a password which without you cannot access the codes.

The phone number here just acts as a username.