People just don't realize that giving someone free reign to their phone is the same as giving someone remote access, basically, to your brain.
It's just hard for the mind to grasp how such a physically small device can grant access to so many things.
Maybe once we move more explicitly to phones as bionic extensions of ourselves, people will understand the access they're granting on a visceral, gut level.