I have lived in the Netherlands and in Germany and experienced the same issues. The natives will tell you that it is due to the language. The language is surely a huge part of the problem but the cultures are also themselves insular. And in any case having to achieve fluency in the local language just to have some semblance of a normal life is too much to ask, imo; not a good deal for expats.
The EU really oversold the who freedom of movement thing. It's never really going to be like the US.