I think people vastly, vastly overstate how well they can actually visualize something in their mind. Most people can't visualize a circle of a solid color and change the color in their mind, let alone visualize a face or anything significant.
People who can draw detailed reproductions are either oddballs with photographic memory, or they've perfected the skill of adjusting their drawings as they develop gradient descent style using detailed recognition as the judge of where to stop.
It is definitely a skill that requires focused training, at least in my experience, so it makes some sense that the average person wouldn't spend too much time going out of their way to practice. If I get out of the habit for whatever reason, I'll slowly lose proficiency at quickly and easily picturing objects in near life-like detail, manipulating them skillfully, rendering complex scenes, holding images with eyes open or under distraction / activity, etc.
Objects will start off grainy/foggy by default and only clarify if I really can focus. Scenes and large complex objects will be much harder to hold together as a whole with distinct parts/areas. Changing an object results in more fog or losing the image temporarily. That kind of thing.
But if the details are unconstrained conceptually, or not tied to a remembered object, then details won’t be consistent as I move around.
The memory palace thing works for me too, if I am not constantly moving things around.