Now that I've said this, I'm sure I'll end telling someone "uh...I'm all booked up" before the week is out. :)
That said, IMO the word "No" is a full sentence, and when clients do as you describe, trying to forcefully argue why it should be a good fit, I simply hold to my original rejection and don't participate in that discussion.
They may want very badly to initiate a game of tug-o-war, but unless I agree to pull on the other end of that rope, it's not happening. :)
I agree 100% with you about the pitfalls of giving a personal critique of why the project is a bad fit/poor idea/not salvageable. That's just bad news.
Also, those bogus inquiries you mention? OMG BANE OF A FREELANCERS EXISTENCE. Not a single week goes by where I don't have an interaction like this:
Small Biz Owner: "I have an app idea!"
Me: "Of course you do."
Small Biz Owner: "You have to sign an NDA before I'll tell you!"
Me: "Straight-up 'no'."
Small Biz Owner: "OK, then...my idea is 'Twitter, but for law enforcement officers'! And they have to pay for it! It's GUARANTEED to make millions!"
Me: "OK, we can build & launch it for $10k."
Small Biz Owner: "What?!? That...ten THOUSAND? That's preposterous!"
Me: "It's guaranteed to make millions, right?"
Small Biz Owner" "Yeah, but...can't you do it for equity?"
Me: "Get out of my office."
(note: the above exchange has been hyperbolized for comedic effect)