I suspect that you may have some degree of aphantasia.
Visualizing something encumbers the same visual processing as sight does. Personally I have a hard time visualizing something and looking at something at the same time. I can only pay close attention to inside or outside, not both simultaneously. (Casual attention, like half-listening, is possible otherwise I'd be a terrible driver.)
To think of it another way: when you say "I've yet to meet someone who actually sees the tree with the same visual clarity as if looking at an actual tree," maybe you should ask some more. I can look at things, close my eyes, and continue studying them up to the limits of my memory and original attention.
Aphantasia seems to come in degrees, not a binary "have it"/"don't have it."