Being cold weakens your immune system. Draft air increases heat loss. There is nothing complex to understand. Outside you would wear a scarf or other appropriate clothing to not feel cold.
Cold, drier air in contact with your mucus membranes lowers your defenses against viruses. It's that basic. In just regular cold air -- not hypothermia.
I'd love for this random thought to be confirmed / corrected.
Now the flu is virus, but it could still have a mind to perceive the seasons in it's infected running state.