>not likely to prioritize a college town as a place to enjoy their years or raise a family.

Maybe. I'd probably argue that, if you want to live in/near a smallish town/city that isn't on the outskirts of a large city, you may find ones with colleges often better than those without. Leave aside the students, there is a significant group of professionals (and alumni visiting from time to time) that can make those towns more interesting than a random small town out in the country someplace.

Of course, tourist towns can have similar although that comes with its own set of problems.