The show Silicon Valley well parodied this phenomenon. It implied that the evasive "no" is commonly because VCs don't want to be known for passing on an investment, outright, that then later takes off.

Which would reflect negatively on their judgement, in more or less a worse manner than it would when risking money on something that fails.

The implied choice then becomes to probably lose money but retain reputation, or to protect money at the possible cost of reputation. Funds can't invest in everything, even if passing on something borderline is a repuation risk. The "not a no but no" third option seeks to both retain reputation and protect money.