I have caught other people's colds a lot of times, but I don't think I have ever gotten sick from dirt exposure.
I once triggered a pretty severe allergic reaction from my partner because of the pollen and dust I unintentionally brought inside after working out in the yard all day.
We make an educated guess, try to collect data in a study that is hopefully a bit controlled, then paint a picture of what we think it means and why things happened (or didn't happen).