That's a pretty wild claim; what observations have led you to this conclusion?
That person may be in your address book, which you've never shared with LinkedIn, but you are in their address book, which they clearly have shared.
No idea how they've gotten access to that, but I've never knowingly authorized a connection to my address books.
The only other mechanism I can think of is that maybe they're getting them via Android or iPhone apps (not necessarily linkedin's app) with less-than-upfront disclosure of what the app's doing. Either way, it's slimy behavior.