You're allowed to do that, but the standard is making no guarantees: C11, § 6.3.2.3(6) “Any pointer may be converted to an integer type. Except as previously specified, the result is implementation-defined, might not be correctly aligned, might not point to an entity of the referenced type, and might be a trap representation.”

I'd consider pointer scrambling to be a pathological case because of that.

Similarly, C11, § 6.5.11(2) constrains XOR to be only valid on integer types, but not on pointer types, further suggesting that you're not really supposed to be doing this.