That's not possible in the iOS version. Tapping on + pops up a request to grant Contacts access.