Wait... What? You don't mind telling Google where your home is, but you mind it tracking your online activity? Shouldn't home address be more sensitive?
It’s not just what Google knows about you. If your account is compromised, attacker may be able to download all historicsl information and share it with everybody.
Is it really unreasonable to expect any company, or Google specifically to decide "you know, home address and travel is more sensitive, let's provide some options to leave that out of ad tracking"?
Of course, I don't think this does much since they can get your address by other means.
Anyhow, giving them your actual home address is like filling out those secret security questions truthfully.
On the other hand, the data about online activity can be voluminous and private.