Yes, I realise that this is part of Android's broken security model.
But the Signal app also wants access to practically everything. Device & App History, Identity, Calendar, Contact, Location, SMS, Phone, Photos/Media/Files, Camera, Microphone, WiFi Connection info and Device ID and call info.
Call me paranoid, but making an app with all those permissions seems kind of the obvious place for backdoors and similar.
If there was a 'light' version of the app which only required access the internet, then I'd be much more likely to install and use it. (And maybe if I ended up trusting it, then later install add-ons / the full version later).