A better test is:
Ask them to imagine a ball on a table, then ask what color is the ball. So far almost everyone I asked answered with a color, no hesitation.
Aphants will hesitate, and they will need to think of a color - in their minds a ball is just an abstract concept - it doesn't have a color unless you consciously specify it.
So far, from my tests - 1 in 10 people asked is possibly an aphant.