It's entirely possible for Signal to work with contacts and yet not transmit address books to OWS. I sketched out a protocol for doing so awhile back: https://news.ycombinator.com/item?id=11289223

So people could use their address books securely if they wanted — if OWS would allow them.