All the reviewer can see is what permissions the app has. They see internet permission and contacts permission, and have to assume that evil code might be uploading the contacts to the internet.
There isn't really any other way to do it - even a thorough code audit (a process that would take months for just a single app) would be unlikely to uncover a well hidden flaw that is intended to allow an evil app developer to steal contacts of users only after the app passes the review process.