The one time I made a change which was drastic enough to consider doing that, I just put a revert button on the interface (on both sides) and got ready to tell people how to use it. It turns out nobody asked.
In general, though, many of the things which are most valuable to test border on imperceptible for long-term users of the site. For example, do you use Dropbox? (Picking a well-known example locally.) Can you identify the H1 on dropbox.com? Can you identify the button copy on dropbox.com? Most seasoned users can't, and won't notice changes to these elements, yet they're strongly influential on free trial signups.
Can you freehand sketch the Dropbox credit card form? How about identifying the button copy on it? These have substantial impact on purchases at the margins. People only see them once (well, typical case for software), and nobody remembers them for more than a few minutes.
Think back to your first run experience with Dropbox. How many steps did it have? What were their names? I'm going to bet you that the onboarding process for Dropbox has had more dedicated optimization effort than everything else the company does combined, but the median number of exposures per user is one.