I don't think it's an issue that it uses your contact list. Why should that be a concern? It doesn't upload them into a database like other services.
Signal is a privacy app, not an anonymity app.
I see people expressing their concern about seeing their boss using the app, etc. and I just cannot relate. Is their argument that they prefer to use insecure SMS with their boss instead? What's the point? Both parties _already have_ each other's phone number as a unique identifier.
I only have a limited # of people I can use Signal with because I don't post my phone number everywhere, though I am old enough that I remember telephone books back when people used to give out their phone number and address and nobody thought anything of it.