I thought I addressed that in my original comment: I _could_ go to each of my contacts and explain why I don't want them to do things like use the cool new Element service with Signal. But 1) I (finally) have a lot of contacts using Signal, so that would be a pain to manage; 2) to me, the entire idea of Signal is that I can pretty much set it and forget it on any relatively-modern smartphone for friends, family, etc. and not have to worry about anything but the biennial phone migration for my mother.
In the end it isn't a huge deal, as most conversations are extremely innocuous, and those I care about I'll take the time to verify. But after all the trouble to proselytize Signal, I get nervous about large public projects that could, in my opinion, strictly reduce the security of my secure messaging system.