"I don't think working remotely scales."
Funny, I feel it's almost the opposite. Remote work can be very difficult for a small business or startup, especially for the early team. Generally speaking, you want everyone within earshot and available for ad hoc ideation sessions, discussions, debates, or projects.
The larger the company, the more likely it'll be organized into discrete functions, work scope, teams, deadlines, and so forth. And hence, each employee will have a clearer sense of what he or she is accountable for on each day, with less moment-to-moment variance. This situation makes remote working easier and more scalable.