The problem, then, is somehow getting rid of the four-wheeler and jeep drivers. (I actually haven't noticed dirt-bikers themselves causing many problems, but they're often accompanied by the others.) As long as they exist they'll be able to lobby the USFS for off-trail access rights to certain locations. In the vast majority of national forest acreage, they are restricted to roads and trails that can handle them just fine.
However, a part of me wants to keep the motor enthusiasts around. They obviously cause more damage than I do as a hunter, so while they are tolerated in places I expect I'll also be tolerated.