There are tradeoffs inherent in every working situation. This is one that really bothered me. It doesn't bother everyone, though.
I find that most talks like that are just shooting the breeze. _Some_ of them are useful and in my experience it's usually then they happen rarely. So I think it is beneficial for the team to get together once in a while, but working like that should not be mandatory and you should just let people go back to their preferred routines ASAP.
UPDATE: And yes, make sure the culture is "remote-friendly" ("remote-first") even if most of your workers spend most of their working time in the office, otherwise it's not going to work (and I'd argue that this is a good approach even if you don't have any remote workers, but that's for another post).