I really like your idea. We're building something similar with a focus on photo sharing.
Have you considered making the contact book upload an opt-in? We also thought about connecting our users via their phone book, but we chose a different approach as we're concerned about privacy.
If you make the contact book upload opt-in, you at least give the user the chance to try out the app before giving away the contact details of their whole phone book (without their permission).