The minute you take away the natural freedom of the big strong man to beat the nerd to death and take all his stuff, you're "structuring" society. After that, you're just arguing about what sort of structure you prefer.

As one of those nerds, I'm happy for the state to structure society in a way where my ability to do things with my brain matters than my ability to fight or lead people in fighting. But its foolish to believe that this structure is the natural order of things.