Really, the only constraint for the semantic of unsafe blocks is that programs that does not contain unsafe blocks cannot have undefined behavior, everything else is by choice.

For example, in theory you can make pointer dereferencing safe (!), and make every operation that might create a invalid pointer unsafe. Rust chose to do this the other way around, probably out of usability reasons.