That there are none left besides the limited interbreeding mark in our genetics could be seen as evidence.
I wonder if the Neanderthal genes we have are more from men or women? Perhaps that would show which direction any conquest and raping happened in.