The purpose of the unsafe keyword is to mark code that could cause safety issues (undefined behavior) if its expectations about the surrounding code are not met. (e.g., an unchecked array access expects its index to be in bounds).
With unlimited time and resources, it should be possible to give a mathematical proof about those unsafe blocks, showing that their expectations are met and that undefined behavior will not actually occur (by for example proving that the array index supplied will always be in bounds).
But since relaxed and consume have unclear semantics, it hard to formally reason about the correctness of any code that uses them (and it is also hard to build tools to do that checking, as mentioned).
So in an ideal world, we would have a 100% clear semantics for all parts of the language, _including_ the unsafe fragment. Unsafe should not be a place to put poorly specified languages features.
Of course, we don't live in this ideal world, but I would rather hope for fixing the memory model so that it be made completely formal and precise (if necessary, giving up some currently allowed compiler/hardware optimizations if they're more trouble than they are worth). As an extreme example, sequential consistency is easy to understand as long as you don't mix it with other types of atomic accesses and avoid non-atomic data races.