Point of unsafe is to say these things _can_ break. Not that these things _will_ break. Calling into unsafe still has to maintain all invariants, so that safe fn wrapping unsafe fn is safe for all possible safe inputs.
Point of safe is unless someone fucked in unsafe, these things will never break.