It does not have to be genetics to show correlation. Sadly research in this area is sparse but we can establish that there will be some correlation;
1) Lack of proper nutrition is linked to lower IQ. This can be fairly easily explained in that the lack of resources will lead to less development in the brain, one of the most resource intensive organs in the body.
2) Certain ethnicity types are disproportionately poor (in the US). Lack of income generally also means that nutritional situation is less ideal.
From 1 and 2 you can easily conclude there is very likely some correlation.
Does that mean we should immediately deploy eugenics? Most likely not. As you mention, high IQ "genes" could be linked to something bad. IIRC they are linked to mental conditions like depression. (Which might lead to an interesting question: if we fix poverty, will people become more miserable on average because the average IQ goes up? It seems likely total misery will go down though)
I also agree with your later statements, a lot of people tend to put all people with black skin into a single pot but there are some many fascinating different cultures and a fascinating history once you go beyond colonialism.
I think a lot of people tend to forget that diversity isn't only skin color. Diversity goes deeper than skin and body.