A lot of people coming in, including myself, are not full time residents. Some of them are retirees from northern Virginia looking for a lower cost of living. Some just prefer rural life with a more live-and-let-live vibe. But almost nobody is moving out here to raise kids who will go to college here.
For the small number who do raise college-bound kids here, a fair number of them will gladly take a lower ranked school to be closer to the city, e.g., University of Richmond.