Human history is unfortunately littered with stories like this, but we know the Neanderthal genetic flow came in from the males. The reason why we know this is we are missing all Neanderthal mitochondrial DNA sequences which are only inherited down the female line. If your theory was correct then we would see modern humans with Neanderthal mitochondrial genomes.
It appears what happened is much more interesting - Neanderthal males mated with female humans of African origin, but the male offspring were infertile.