The way they work is based on the idea of a narrow stream of air that gets water off your hands better. But for the stream to remain narrow and not dissipate, you have to get your hands close to the equipment. It's based on almost but not quite touching it.
Positioning your hands in a careful and controlled manner is, ergonomically speaking, an uphill battle. So on average, it won't happen often.
In other words, Dyson hand dryers kind of trap you into accidentally touching the part you don't want to touch.
Worse, the same applies to all the people who used the hand dryer before you. It's a very reasonable bet that the surface where the air comes out has been touched by someone who didn't use soap when they washed their hands.