We don't even keep very close to that as it is. Between daylight summer time changes and some time zones already being very far off from geography, some places like western Spain or China are already as much as 3 hours off from actual solar time.

As a sibling comment says it already differs by about 30 minutes over the a year. And it will take roughly 4000 years for the mean solar time to drift about an hour--- so thousands of years for the mean time to dominate the error.

This makes getting a fairly accurate solar time arguably easier, in fact, because you can use a formula correction on top of a stable timebase without first backing out leapseconds which may or may not have been correctly applied.

Solar noon already varies by up to 30 minutes throughout the year, due to axis tilt and elliptical orbit.