But that shouldn't be a problem for tardigrades frozen in a block of ice, because they are not in direct contact with air.
I would suspect that some of the oxygen from the water is liberated occasionally.
A quick google search didn't turn up anything, but I'd suspect that water spontaneously liberating oxygen is a very rare occurrence.
Oxygen is in a high oxidation state and once reduced to water is quite stable. In order for oxygen to form from water, you'd need an oxidizer of considerable power (e.g. ozone?).
Perhaps, although oxygen is a quite heavy element, and doesn't jump around in ices as much as hydrogen does.