What you're referring to as "the server" is really a technology stack that starts with the CPU hardware and extends to the service operator. Without SGX, the trust stack probably looks something like: Intel, the OS, the VM, the cloud provider (Amazon, Google, Microsoft), the server software, the server operator.
Any of those points on the trust stack can be compromised by their principals or by an attacker (for example, Amazon could choose to interfere with Signal, or someone could hack Amazon and do the same). This is true for the OS, the software, the service provider, and everything else in the stack. Users have to trust all of those points, and additionally trust them not to be compromised.
SGX shortens the trust stack all the way down to the CPU. It takes the OS, the VM, the cloud provider, the service operator, and everything else out of the equation. Clients no longer have to trust any of them, either to be honest or to avoid compromise.
If trusting the CPU isn't acceptable to you, then that might be a bigger problem than you realize, since all computing has to. If you'd rather put your trust in AES, the problem is that you still have to trust the CPU to perform AES the way you would like it to.