If you crash your own party with some people that don't fit in, you will cause the guests you wanted to have to leave.
Any countries not in the EU don't want to be in the EU. And that's fine.
And looking at what the EU does you don't have to be scared if you don't join. It's a defensive union first and foremost.
Switzerland isn't scared about the EU invading despite largely uncontrolled borders.