This is why "2FA" is supposed to actually be two factors. If you're using a phone number for 2FA, then authentication still boils down to the same thing: Something you know.
It's the password recovery by phone that's the weakness. But I think people getting locked out of their own account is probably a bigger problem for Google than people getting hacked, so they err on the side of saving your from getting locked out.