I've done it when we really needed that performance for an inner loop(particle system). It can be a real bastard to keep the non-alias constraint held constant in a large, multi-person codebase and the error cases are really gnarly to chase down.
Compare that to Rust which has this knowledge built in since it naturally falls out of the ownership model.
At least you can audit only those places though.
That's why Rust developers don't take kindly to unnecessary unsafe usage, because the audit surface area (and interaction complexity) increases.
Therefore, the optimizer can always assume mutations don't alias. Even UnsafeCell isn't allowed to break the alias rules, it just provides flexibility going from immutable to mutable.
For example consider Slice::swap [1]. This does consider the possibility that the pointers alias, and correctly, because they might point at the same object.
https://doc.rust-lang.org/src/core/slice/mod.rs.html#541-553
LLVM implements only coarse per-function aliasing information needed by C, and doesn't properly preserve fine-grained aliasing information that Rust can provide.
That said, there is a series of patches in-flight to get actual working full restrict support: https://reviews.llvm.org/D69542
And how do you know this? Have you actually measured the difference?
My CPU-heavy program (which takes ~30 seconds to run on a 32-core Threadripper) gains ~15% extra performance if I turn it on.
That's like saying C programmers could just write memory safe code if they felt like this would help, so clearly memory safety is not important.