You may not be wrong.

It costs a lot to move. Not sure about US but here we have stamp duty. For me it's a year's salary. Another half years salary for agent fees. A month's salary for removalist costs. The only way to make it feasible financially is to do a reno project but that's more stress.

Selling stuff and being creative with space may solve issues. Living with "lower class" people isn't necessary a bad thing it may be good to prevent living in a bubble.

Yeah moving is stressful. If you do go ahead I wonder if there are services to manage everything for you. If you have the money maybe get the new place furnished ahead of the move.