I don't disagree, but on the other hand: Lloyd's used to be a London pub where merchants would hang out all day, carrying out their business as they ate. Business deals at the highest level are still routinely done over breakfast, lunch, or dinner.
I think the problem, as usual, is the power imbalance: they are eating and you are not, in addition to the fact that they have the money and you don't - so it becomes a bit too humiliating.