I actually just checked my address book and it seems I have never saved his phone number as a contact! The only other explanation could be that he saved my phone number as a contact and let Facebook have a field day with it