I always opt-out too. It's a matter of principle, I refuse to have my privacy invaded like that without being able to see who invades it, plus it creates more work for them.
Also, I rather enjoy pat-downs. They're like a massage, and the TSA agents wasn't too overt with the problematic regions, although they did try psychological manipulation to make me feel bad (refusing to let me touch my stuff, making me take my belt off, etc). I didn't mind, though, opting out felt like a big "fuck you" to the TSA.
Of course, I don't even live in the US, so I've only done it twice.