That's not the case. Children can learn how to drive in snow or dirt roads in a matter of hours. Learning to drive on public roads has more to do with learning the rules than learning to drive a car per se.
Any folks who grew up on a farm who can confirm?
I wasn't allowed to fell the trees but I had to use a chainsaw and cut the branches off the felled trees and then section the denuded trees into small enough rounds they were suitable for busting into firewood with an axe. I was given the option to learn to drive my grandfather's stick shift Toyota pickup truck in order to move all the wood with that, or move it by hand. It was an easy decision.
But, yes, until you're talking about more serious off-road/bad road (which is challenging in a completely different way), there's nothing especially hard about driving on a private dirt road.