They do a lot more than that - e.g. they scan all your contacts so they can spam them as well.
To this day, I still have ex-girlfriends, one-time ridesharing buddies and even some fictuous people "waiting to connect with me" on LinkedIn.
http://www.imore.com/how-restrict-apps-access-your-contacts-...