The evidence I have is the dramatic drop in perception quality when my eyes are closed. Even though my minds eye is exceptionally vivid, there are differences when visual data is not available.
3d stereo reconstruction of surfaces is one area that is greatly hindered. When my eyes are open, saccades cause waves of stereo correlation to pass over an object revealing it's surface geometry in extreme detail.
Ongoing querying of reality is useful to explain some other detail related perceptual phenomenon.
There was also a visual perception demo I saw at OMSI a few years ago in which a scene had objects appear and disappear. A frame of black between the changes completely disrupted normal change detection and it was very difficult to detect which objects changed when relying on only "visual memory".
My evidence is empirical, but I'm sufficiently convinced that our minds use reality directly for parts of our working memory.