Well, it's true. But if you're paranoid you choose not to believe it.
Except for some smart alecks who would put in "Candidate held fork in left hand, was neat eater" I never saw anything beyond "n/a" or similar non-feedback. Most often I would see a savvy recruiter pair up the lunch with the referrer so they could catch up with their friend and it would act as the 'sell' part of the interview slate.
He joked that "Well, if you punch a hole in the wall I might write something". Other than that we just talked about hacking game consoles & Chromebooks & breaking out of Chrome's sandbox to reach the OS from javascript.
This is a functional and smart practice.
I've definitely seen the lunch interviewer pulled into the conversation. When the process isn't supremely formal, lines get blurred. Being paranoid is probably a smart thing to do.