The we you're referencing includes very few people in the world.
If so many people are truly free to pursue higher callings, why don't they? Look at reality, not theory. Find some answers to that question and you'll start to discover the wide and complex range of restrictions on true freedom.