Even for the people who didn't eat with the group every day? I notice that those people tend to get excluded over time. Not on purpose, necessarily. It's human nature to see things in terms of "part of the tribe" and "not part of the tribe".
> maybe you're not working with the right people..
This depends on a lot of factors. Even if you absolutely love everybody you work with, you may not want to eat lunch with them every day. For example, you work in an open office environment and see, hear and talk to them for 8-12 hours every day. Some would argue that, under those conditions, taking an hour in the middle of the day to be by yourself or to meet with people you don't see quite so often would be considered healthy.
On the other hand, if you all work in private offices 8-12 hours every day, then taking an hour and socializing with your co-workers sounds fantastic.