Well, the United States is a living example of a country that tried that, and um ... hasn't failed because of it.

Until the Chinese Exclusion Act of 1882, there were no restrictions on immigration. And after that, the immigration of white people was still completely unrestricted, until the Immigration Act of 1921. And after which, it was still relatively easy for citizens from Western European countries to move to the US (thanks to the "National Origins formula", up until the Immigration Act of 1965 was passed.