Google could request permission to access my contacts, which would give them access to my contact card, which has that information...

Instead they want me to sign into Google and it wants that info to then be stored on Google's servers.