> as long as you don't touch NaN

Note: this says NaNs one place will be NaNs everywhere, just the precise choice of NaN bit pattern is unspecified.

So there is no issue “touching NaN”; the issue would be trying to pack some additional information into the fraction bits of the NaN, and expecting every implementation to treat it the same.