Unfortunately I think you're underestimated how uneducated some of the population is. I think everyone knows it's not dairy, the confusion comes in if it's nurturing like milk. When people say "milk" the first thing that comes to mind (usually) is a mother's milk (any mammal.) So when they see "almond" and "soy" milk they probably think they are equal alternatives which just isn't true.