As a remote worker I've seen it done well and I've seen it done poorly. I think you have to go remote first, with all meetings and communicating being inclusive for the whole team. If you decide something face to face, send a summary via slack or email.
Anyway details aside, it can be done, it's just it has to be a deliberate effort from everyone in the office.