I find that there are significant benefits to working in the same physical location.
It's just way easier to go to someone's desk and ask them something than to have to set up a video chat. Human interaction is just naturally an in person thing. So much is lost if you can only video chat. Sure it may not affect the work directly but I think it really affects a team's dynamic. Lots of great ideas happen just from random office interactions, whether it be lunch or just talking over coffee. These kinds of things are really hard to reproduce in a remote environment.
That's not to say there aren't reasons working remotely is beneficial. I just don't think it's fair to just say the only reason companies opt for physical proximity is cultural.