I don't think it's as simple as what you're saying. We have evidence that 2-4% of the entire modern genome of non-Africans came in from Neanderthals, and an even larger amount of genetics coming in from Denisovans for many in modern Asia and beyond into Australia.
As I understand it, all Neanderthal genetics that modern man has came in through the X chromosome. This could indicate that male offspring of Neanderthals and Homosapiens were infertile, which would be a cut-and-dry indicator that they're two separate species.