Use oil on your hands before dealing with hot peppers. This is second nature in places that use a lot of spice.
Gloves anyone ?
I am never able to just "wash my hands" after the good peppers without additional precautions like oil or gloves. Jalapenos? Sure washing takes care of it. Habaneros and significantly hotter to the far end like Carolina Reapers? Yea, gonna have to be increasingly careful.
Generally, it's not the hands that are the problem, it's when you touch your eyes...or other body parts that the capsaicin transfers to that makes things unpleasant.