No. The clients are open source, and (at least on Android) you are able to verify that the source on Github is the same that was used to compile the client on Google Play with reproducible builds [1].
And even if the servers turn out to be malicious, the clients are designed to expose as little metadata as possible with things like private contact discovery[2], sealed sender[3] and private groups[4]. It's not perfect, but the data a malicious server could collect is limited.
[1] https://github.com/signalapp/Signal-Android/tree/master/repr...
[2] https://signal.org/blog/private-contact-discovery/