This is exactly it. LinkedIn harasses you to turn this on every time you launch the app on your mobile. If you have their number in your contact list and you enabled this setting.

I don’t know about Android, but in iOS you can see if you have enabled access to contacts. It’s also totally feasible your contact info is in their phone and they enabled it (email address).