Why do we adjust time, and not geometry (given that geometry changes anyway because of shifting of tectonic plates)?
(By geometry I mean coordinates of objects like buildings, landmarks, etc.)
That’s more than 1000 times slower than leapsecond-related drift
We do, however, take plate movement in to account when determining positions of things on the surface. Geodesic coordinate systems periodically publish 'Datum' that are the result of new surveys of positions. In some places, such as Australia, the difference a decade can make is pretty large. Other places, not so much.
With your proposal, time handling would be slightly simpler, but every geographic coordinate everywhere would need to be tagged with a date to let you know which epoch it was relative to, or else you would get huge (in local terms) errors.