Clearly people who want to go to an office should join companies/teams that go to an office and people who don't should join companies/teams that don't.
Or there could be, ya know, a mix. Again, you are trying to impose either or, whereas those of us who work from home want to be accommodating. Perhaps because we, despite working just as hard, are a bit more laxed in how we view the world around us.