Stone and adobe, in the right conditions.
https://en.m.wikipedia.org/wiki/Ancestral_Puebloan_dwellings
https://en.m.wikipedia.org/wiki/Ancestral_Puebloan_dwellings
ie, the durability of those structures is more due to where they are, than what they are made of.
Something like granite would be a much more long-lived structural material in wetter climates. Indeed that's the primary stone used in, say, Machu Picchu.