I would think it should be possible to make a decentralized back-end where servers don't need to be trusted.
This is trying to combine 3 points on Zooko's Triangle [1]: You want human-meaningful names (which phone numbers are, because they map to existing things), so you have to make a trade-off between decentralization and security. WhisperSystems opted for security for some reduced decentralization. For something that’s aiming to replace text messaging, I can’t really blame them for that choice.