No, that is not the point. You are putting words into Chesterton's mouth.
In his scenario, 100% of the people entrusted to road and fence maintenance considered keeping the fence and concluded it should be torn down. The people that installed it didn't think it was important enough to document its reason for existing.
The only person advocating for its continued existence is a philosopher with no apparent expertise in fence maintenance, or any specific knowledge about the fence in question. He is demanding the proponents of tearing down the fence produce evidence from the known-lost historical record, and then to use that evidence (and only it) to litigate for removal of the fence.