As is usually the case, reality is to messy for simple explanations like this to be completely correct. Europe and Africa are pretty close, so there has been a lot of migration to and fro, some of it bringing Neanderthal DNA into Africa: https://www.sciencemag.org/news/2020/01/africans-carry-surpr...