This would explain contact alleries to for example nickel and various chemicals, which are usually quite harmful, but I don't see how this would explain pollen and animal allergies. A lot of the pollens that cause allergy in some people are coming from harmless plants. And with pollen spreading in the wind, it would be an annoyance without necessarily protecting people much against coming in contact with the plants. Would avoiding contact with certain animals be beneficial? Other than avoiding pathogens, I don't see a lot of reasons to do so.

Anyway, I wouldn't be surprised if allergy, like many things in biology, are caused by multiple factors instead of just one.