A college town is, really, only desirable to college students and the businesses that cater to them. Thus, the demand is very low, and almost nobody stays particularly long. The exceptions to this are colleges in or near very large towns, but even housing near the college tends to be much cheaper than housing in the city.
If McD's was a self-contained economy, it too would be surrounded by cheap housing.
I wish we could have conversations about things like this which are based in reality and not some fear of a macguffin enemy trying to "keep the poor from existing".