Each virus can only interact with certain cells of your body. COVID and influenza interact with lower respiratory cells. You have to inhale a viral load to be infected, or at least this is the consensus belief right now (and is contrary to all of the concern about fomites early on).

Many cold-like viruses infect mucous membranes (you have a number over your body). A glancing touch is enough to be infected.