What platform are you on? I use signal on android. I declined it access to my contacts on initial installation and it hasn't asked since.
You're a bit disingenuous in the way you seem to be handing out unwanted and unwarranted moral guidance based on simple questions people ask.