The weakest link is some of your contacts that opted (because that is the default) to upload their phone contacts to a Facebook service. Through those phone contacts, Facebook correlates the contact name with your Facebook account.
Ah fuck. I was wondering about this because I was in the same boat as OP. Even weirder - I checked my profile settings and my phone number was listed even though I'm certain never added it and stopped using mobile years ago.