You are actually right, historically speaking. After the United States victory in the Mexican-American War, there was a movement to annex all of Mexico.[1] The decision to not do so was based on racial concerns. Senator John C. Calhoun of South Carolina who opposed the annexation of Mexico, made this speech to Congress:

We have never dreamt of incorporating into our Union any but the Caucasian race—the free white race. To incorporate Mexico, would be the very first instance of the kind, of incorporating an Indian race; for more than half of the Mexicans are Indians, and the other is composed chiefly of mixed tribes. I protest against such a union as that! Ours, sir, is the Government of a white race....

[1] https://en.wikipedia.org/wiki/All_of_Mexico_Movement

You are very right, especially places like colonial Africa. However in the context of California, Arizona, etc much of the immigration is from Guatemala, Central America, etc, not the true native tribes like the Chumash, Luiseño, et al who are still here.

Just pointing this out because "division" gets bandied about a lot inaccurately.

All to protect mafiosos who pretend to be organizing "unions" for the "workers" benefit (which is illogical if you think about it due to the Principle-Agent problem, and makes no sense economically- the bosses will cut deals best for the bosses, not for the union membership.)

Keeping talented people down is just a side effect.