Why would anyone get your contacts if you sync them via your own CarDav server? You don't need to sync them to iCloud / Siri on iOS, and you can use non-Google Android phones.
Whatever you're using to sync contacts, without syncing them to this contact app, it seems (to me) as useful as keeping a textfile full of phone numbers in Dropbox. You won't have any useful prompts telling you that it's Dave phoning you, John texting you, or Alice's number to phone/text.
This is, admittedly, neatly worked around by keeping your permission requests under control, but... stuff like this[1], from before the granular permission system on Android was available, are annoying.
1: http://bgr.com/2011/08/12/facebook-stole-every-contact-and-p...
But that wasn't the article's concern at all, as you pointed out in another comment.