>I don't like the idea of building my entire friend group, or a majority of it, around who I work with.
I think you're seeing this in the wrong light; you interact with your teammates (even remotely) way more than most people. Isn't it healthy to have good relationships with them?