Effectively it is an atmospheric interaction, most commonly with oxygen but also light, hydrogen sulfide (from agricultural production - land is cheaper in the countryside), and other sources of contamination.
Most wafer processing - especially depositions of materials like metals - are done in a vacuum. if you deposit a metal (lets assume aluminum) in a vacuum you effectively get a pure aluminum layer. Pure aluminum is highly reactive with oxygen
As soon as that layer is exposed air it oxidizes - same thing as rust. You know this as the dull surface finish that raw/bare aluminum gets if left out or when you purchase it. It's what happens very quickly after you machine aluminum - the shiny surface goes away. That is the aluminum oxidizing.
Aluminum oxide has different electrical properties than pure aluminum, different enough to fundamentally affect the function of the transistor devices being constructed. If I have a 2" thick piece of aluminum that layer, maybe only a few microns (assume 5) thick after 24 hours, is basically inconsequential. It represents 0.01% of the thickness of the piece. It is error but not much more in the electrical properties of the piece. However, if semiconductor layer is much thinner... maybe 10microns thick...the effect of the oxidation on the electrical properties is much much higher and pushes it outside the tolerances the device can have and still function.
And because the deposition layers aren't just surfaces, but penetrate to fill trenches in the devices they are basically impossible to remove at most steps.