It makes no sense to "get fully rid of unsafe" and this suggests you've gravely misunderstood the problem. Which puts you in good company, Herb Sutter doesn't seem to understand this on his "CppFront" wiki and Bjarne doesn't seem to grasp it in his recent paper about safety either.
Rust's unsafe keyword marks code which programmers intend to be safe but the machine can't see that. For example the Rust compiler can't see why the Linux implementation of Mutex<T> is correct, why would we give out mutable references to anybody who calls this function named "lock" ? The programmers (in this case mostly Mara) know how the Linux futex system call works and their reviewers have concluded the resulting unsafe stanzas, with their commentary, are correct. There will in fact only ever be one mutable reference at a time even though the machine can't see why.
The reason to care so much about memory safety is that you can't have type safety without memory safety, and when you lose type safety most of your other guarantees are destroyed. Languages which claim to care less about memory safety often have a caveat (even if unstated) that all bets are off once you abuse their lack of memory safety to destroy type safety because all their other promises assumed type safety and now they don't have that.