The problem as I see it is that having a &mut reference to something lets you move it via mem::swap/replace (and maybe a few others?) -- but actually needing to do so is rare. It seems to me that if that weren't allowed, taking a &mut reference to a self-referential value would be perfectly safe.

Maybe there could have been a way to opt in to moving-via-reference for those rare cases when you need it. Or maybe this whole problem could have been avoided by making swap and replace unsafe? I'd love to see someone explore that design space.