The structural integrity of (comparatively) smaller fossils is much more likely to survive for 100s of millions of years. A large building will be eroded by roots, wind, acid rain, etc. A fist sized rock containing the imprint of a trilobite, buried in soil and safe from most elements, will easily survive for 100s of millions of years.
Or perhaps the author meant that the vast majority of our buildings are not built to last. An asphalt highway - if unmaintained - will be unrecognizable in a hundred years. Roman roads survived for a few 1000 years because of their very crude nature: they are just a pile of rocks. But even such structures will fall prey to soil erosion and the relentless assault of nature. Unless said roads see foot traffic, they'd be also quickly overrun by trees and grasses.