Without a government they can start to take this authority themselves in much the same way that local warlords do. Gradually by force, because they have power and you don't.

A small / weak government can't do anything to stop them.