Having rich countries support its poor neighbors is an ingenious solution to improving your quality of life. You impose your rules, regulations and monetary policy, they get capital for internal improvements. If there's no huge waste or theft (which sadly exists), you end up with wide, strong and stable continent-level middle class. Which is great goal, as we can see when observing Switzerland -- wide, strong and stable country-level middle class.
Last time Switzerland attempted something like this (~10y ago), it got burnt, hard (lost a lot of EU related projects and academic financing). Cutting the economical/market ties with the EU, considering its position and dependencies, is a suicide.
This is entirely about free movement and immigration.
What the EU needs to get rid of is of the veto power. Otherwise I welcome our neighbors to the east as long as they are willing to play by the rules.