The protocol has a good reputation, problem is that you can't tell that the app is identical to the source, and each new update has the possibility of breaking things.

Many people were angered by the recent update that forced users to have pins, which many didn't want. Doubly so since there's no recovery method.

The pin issue highlights that future updates could make things less secure and/or add backdoors. Various legal efforts in various legal jurisdictions are trying to enforce backdoors and it remains to be seen if whisper systems will pull out of a jurisdiction if forced to compromise on security.

I can't speak to the iOS client but the Android one is only technically OSS in that it bundles proprietary binary blobs by default.

Who cares if the server is not ?

OSS /= no backdoors.