There’s no easy bound on the set of code you’d have to audit to confirm that even one use of unsafe is in fact safe.
fn do_something() {
unsafe { ... }
}
// Somewhere in the program
do_something();
Doesn't matter where "do_something" is used and how much. The only possibly problematic part of this code is the unsafe block. You only audit it.