(If you're allergic to something more exotic, you also know to not blindly trust the reassurances of people who, while good natured and friendly, never had reason to learn the fine points of allergy stuff.)
Also, people don't generally identify with their allergies, it's not like acknowledging the existence of allergic people in a redundant way validates them or something. A sign notifying that milk-free options are available is plenty.