Granted, this is pretty hard to solve when participants come online and offline at different times without having a trusted and always-online entity to handle the list of the current members (in the signal model, it's the server), but signal's still definitely not a silver bullet, even if people treat it like it is.
But if Signal gets pwned or captured, it can easily add itself into any group, or even add and remove instantly.