I've worked at similarly (or perhaps even better regarded) companies with lunch interviews. They always say the same thing.

I've definitely seen the lunch interviewer pulled into the conversation. When the process isn't supremely formal, lines get blurred. Being paranoid is probably a smart thing to do.