Signal explicitly considers anonymity/a private social graph to be a design goal. Their documented considerations of things like sealed sender and use of Intel SGX's remote attestation clearly demonstrate this.
Yes, these measures are imperfect. Like you say, it's a centralized service and has weaker anonymization properties because of that. However, because of its design, to de-anonymize Signal users requires the server operator to do continual, non-trivial work as long as they want to de-anonymize users.
In other words, deanonymization attacks against Signal users works just like any other kind of attack: the question isn't whether it can be done, but whether the attacker has the resources to do it. Presumably you'd consider Tor to be an anonymity tool. Yet, even though they're distributed, deanonymization attacks against certain users can succeed, given enough resources. (I don't mean to imply that Signal is equally resistant as Tor!)
Note: big fan of the work you all are doing at Amutable! I'm following the news eagerly. Best of luck!