Using the US as an example, in the 1800's and early-to-mid 1900's, American work was largely labor intensive agricultural and factory based work. Labor was in huge demand, and as a result the middle class was able to thrive. Today that is no longer the case, and a massive rift between rich and poor is developing as the middle class collapses without an economic foundation to support its existence, this being the labor intensive industries that have largely disappeared in modern Western nations. This has been the development path of all developed nations.
As modern economies are already struggling with worsening labor conditions for the majority of their people and the resulting stresses on social programs, it makes little sense to open immigration to the impoverished masses.
This is a realist perspective. Denying the economic factors at play here will only lead to increased socio economic tension in developed societies at a time when such tension is already at a high water mark in modern history.