> The fallout people keep missing is the loss of tax revenue. Either the direct property tax as values drop to in some cases income taxes shifting to other locales. Throw in the reduction in spending at local associated businesses like restaurants and stores which also results in lower sales tax revenue and eventually staffing. It is quite the cascade effect that will happen.

I think we might see the opposite in some subset of small towns - probably the ones in and around areas of natural beauty, or hot-spots for hobbies like rock climbing or skiing. I'm guessing some places are small enough that it wouldn't take a huge number of well-paid remote elite moving in/nearby to boost local government coffers, and possibly make some businesses viable that weren't before.