I agree that the latter is indeed a pressing issue. But my point is that the former isn't about unsafe code so much as safe code: safe users of a concrete pinned target type may observe any number of different API surfaces, some of which allow modifying or swapping out various parts of the pinned object.
For a practical example of this, when the target type uses one of the pin-projection crates, it has a choice in which fields to denote as structural or non-structural, which affects how much the user can modify each field down the line. Indeed, a type could usefully choose make all of its fields non-structural, if only its address is externally referenced.
The pinning invariants simply don't fully constrain what the user of the target type can and cannot do, except for those constraints needed for the target type's own soundness.