To put this into perspective: I used to have a webpage which ranked quite highly for searches related to chastity belts. Do you think the average visitor would appreciate me knowing the exact search terms that landed them on my page?
Honestly, I trust Joe Random Webmaster a lot more than I trust Google, because some small site manager doesn't already have a vast corpus of data about me.
At least assuming none of their employees goes very rogue and starts leaking data en masse, or has a personal vendetta against me or something. I know something like that happened once (https://www.wired.com/2010/09/google-spy/), but the odds are probably still pretty low.
Also, though it may be naive, I believe Google does not intend to cooperate with the NSA any more than they're absolutely legally required to.
The NSA aren't the only parties attracted by Googles large Data are Liability stash.
I agree that's the biggest risk.
Almost all the referrer logs from Google Images had search terms like "hot under 13 girls". I banned the Google Image robot from the site, since no useful visitors arrived this way.
By concealing the keywords people were searching for to land on your website, they force content hosters and ad networks to depend on the Google analytics + ads ecosystem. With these keywords, you could gather information about your Google rank for queries.
Whether it's net beneficial for you as a browser owner to pass referrer information - probably it is if you like the people hosting the content you read, as it helps them understand their traffic and improve their service. If you don't like them (probably because they are spam) then this just helps them be more effective spammers.