As I understand it, their mitochondrial DNA does not survive, so they were always the fathers of any hybrids that contributed to the modern human genome.
" recent Science study found evidence that couplings followed a distinct pattern: Neanderthal males with modern human females."