And indeed the planning board can make an exception:
A private road, but as with Class VI roads, only if the governing body, after consulting with the planning board has adopted a policy allowing building on that particular private road, or portion thereof, and then only if the owner has recorded a notice in the registry of deeds acknowledging that the town is not liable for maintenance...
So I would think that a farmer that wants to build a storage shed on the far side of his field would be able to get a permit as long as it's accessible by some private road (even a class VI dirt road)