Very good question. That's probably one of the largest sources of division between various anarchist schools.

One way to tackle the problem is to separate governance from the state, and say that we want a governance structure accountable to the workers, and populated by workers, but no associated state apparatus—no police, no army, no state lines, etc.

Many forms of anarchism turn into a kind of pastorial agrarianism because of this. It really is hard to come up with a governance structure beyond a certain size that meets the accountability requirements for the kind of society socialism wants, and so many socialist anarchists see the answer as the self-governance of communities small enough to manage some kind of general council.