I know exactly what you're talking about.

I say No a lot. It's not fun to say No, but I just try to be honest. When you get to have these conversations a lot you gain useful insights... there are a lot of ways to help people other than building their product for them.