Modern humans have very low genetic diversity. There are more genetic variations among apes in a single group than among all humans on the whole planet. A typical interpretation of this is that we are descendants of small group of survivors, but whatever the reason the consequences is that the genetic variation between Neanderthals and humans can be in fact within normal variation within single species, we just do not have enough data.