They have a privacy policy https://whispersystems.org/signal/privacy/
They have explained their reasoning https://whispersystems.org/blog/contact-discovery/
You have access to the source https://github.com/WhisperSystems
You have proof that they don't store your contacts and can't provide them even after a subpoena https://whispersystems.org/bigbrother/
Again, what exactly are you nervous about? What is gained by removing contact discovery?