Yep, 2-4x onsites together a year to develop human relationships, and otherwise 100% remote, is by far the most effective team arrangement IME. It is especially the most accessible format for people who do not necessarily perform their best work within the typical office hours (9-5, +/-1), who do not want to live in your metro area, or who are distracted by disturbances in their surroundings - or just aren’t hardcore extroverts.

Or simply put: if you truly want the best, most focused, highly performing team, an office requirement shrinks your talent pool tremendously for extremely little gain. Do quarterly meetups somewhere and move on, IMO.