I rarely travel alone; it's usually with a wife, and soon it'll be with a wife and kid, so I'm not sure we'll get the option to just say no :/
Or, say, turn around and stay where you are instead of going home.
I do get the pat down pretty regularly, but I can't say for certain that I'd cancel an entire flight (and take the hit to either my professional reputation (outbound) or delay getting home to my family (return)) to stand up for this. That's not even considering the financial hit - I doubt the airlines would be sympathetic and waive the $100+ rebooking fee they've started charging.
I get the principled stand you're trying to make, but there's quite a few gray areas here.
edit: balancing parentheses, and context
We were each assigned, simultaneously, two employees (so four people, total): one to do a pat down and the other to just stare at us and lord over our stuff. We were then asked a barrage of questions about who we were, why we were in the area, who the other person was... great questions that were absolutely hilarious given my job and the two of us being a tiny tech company ("what is his job title?" "oh man, what does he call himself? I think it is Vice President of Business Development or something like that?").
I was asked about how I felt standing in line and then the interrogator used my answer (which I will state was extremely deferent; I actually can tell back most of the interaction pretty accurately, but I will save you that version of this ;P) to kind of chastise me (we were late for our flight, by the way, and knew we had no way of getting to it in time, but simply did not care as we had nowhere to be in life for days and we both seasoned and crazy travelers, so I wasn't at all rushed or bothered). I hate these people. I hate them so very very bad.
Ok, but so: in my experiences with them I think you will just fail. The TSA agent will be quite happy to have you sit there until you miss your flight, and they will probably love telling other TSA employees stories about the asshole in line who thought they would stand up and fight but who got no sympathy from anyone in the command hierarchy and got to sit there "as I smiled at them and shook my finger" as their flight took off :/. Or maybe I don't know what your goal is with "turn around and stay where you are" (actually, after typing all of this, I am wondering if your point was that you might get stranded when you are out and about?).
TSA agents run the gamut from fascist jerkwads to rando people working a crappy, thankless job. I'm sorry you got treated badly - I've also had bad experiences going through opt-outs.
As I've said elsewhere in this discussion: You can't win the war by losing battles on the ground. You have to take the fight to the top. If you want change, you have to approach it politically, as disgusting as that concept is. Unfortunately, I fear it will take an entire new generation of Congress to effect that change, and with no term limits that will never happen.