Grew up with copious amounts of Norwegian cow milk - and zero allergies. :) Back then we were free roaming kids as opposed to today, where I now am in Melbourne Aus and it's very very different. However, neither of my children have picked up any either.
The fear for peanut allergies was so big that we introduced peanut butter to them at GP's office with epipen at the ready (very early in their lives). Feels a bit crazy, but that did get them introduced safely and early.