Hallucination and visualization are distinct things. People without aphantasia who talk about "seeing" things in their mind are not confusing the mental images with the scene in front of them, that would be a hallucination.
From the way non-aphantasia is characterized here one could assume that the difference is just how much control people have over the content of the image, as opposed to the degree of realism.
A hallucinating person may have an experience that feels indistinguishable from reality, but they can't control what the experience entails. From this test, I gather that a non-aphantasiac person has an equally-as-realistic image in front of them, but they are completely in command of what is shown.
The test asks me to visualize the face of a close friend or relative. I can quite clearly bring to mind my wife's face. I can imagine looking at each individual mole, or different facial expressions she makes.
If I were asked to visualize the face of the barista who served me coffee 20 minutes ago, I could only come up with something vague. I remember he was wearing large earrings, because they stood out to me, but his face is a blur. I mean that literally: when I imagine looking at his face, there are parts that simply won't come into focus or even into view, like they're missing -- in the same way that the dot disappears when you find your blind spot (i.e. not in a "argh, he's missing a nose!" way, but in a "it's just not there, but that's not weird" way).
So I have a pretty clear phenomenological distinction between visualizing things clearly and not.
It was in response to the complaint that the "Vividness of Visual Imagery" test is ambiguous, because people can't decide whether their visualization is clear or not. I'm saying that, for a person with a good ability at visualizing, the distinction is fairly clear.
And it's not just fuzzy memory, the fuzzy memory causes a non-clear picture. But I can't transfer my phenomenological experience to you, so you'll either have to accept that some people can have both clear or non-clear imagery, or not.
In answer to your question, I can certainly produce a mental image of an imagined barista. I can imagine very fine details in, say, the handlebar mustache I invent. Yes, those details will probably have come from reality, but I don't know from where, and I can visualize it very clearly if I choose to.
In answer to a question you didn't ask, when I read books I rarely have a detailed image of a character. They're kind of faceless people, roughly sketched. I actually discussed this last week with my either-year-old, and she had the exact same experience, but had never really stopped to think about it. (Like what I was saying before about the nose missing not being weird.)