Yeah, `unsafe` is the developer signing a contract to uphold all the guarantees that safe rustc provided (at least from the perspective of the public interface, the invariants can be broken in private code).
That's why Rust developers don't take kindly to unnecessary unsafe usage, because the audit surface area (and interaction complexity) increases.