Logistics. Crested Butte is 3 hours from the nearest larger town, Grand Junction. Assuming good weather.
You need workers and materials to build new housing. There is already a shortage of workers.
Builders in Grand Junction can make far more profit by building in Grand Junction, vs driving 6 hours each day to Crested Butte and back. Same with hauling materials.