Imagine, one of your contacts is captured; attackers get his contact list that includes you; then they get your phone number from Signal; then they get your location and put you to all kinds of black lists, extremists lists, no fly lists, watch lists and so on.
Signal is nothing better than Telegram. They should be on the same position.