From the stores perspective, I can imagine that there are a few reasons why it can make sense for them to move out of impoverished areas even while demand exists there.
Wealthy customers who can easily travel to stores in impoverished areas are turned off by poor conditions and higher crime rates and will shop elsewhere. They may also feel less comfortable shopping in areas where the majority of the other customers are of a different race or culture. The stores and staff may themselves experience increases in theft, vandalism, and violence in low income areas. Stores can't sell as many overpriced items in low income areas where the majority of their clientele are struggling to afford necessities.
It really doesn't matter that a store can make a healthy profit by serving customers who live in a low income area. If that same store feels that they could make even greater profits by being in an area with wealthier customers, then that is exactly what we must expect they will do.