Some time ago I discussed the idea that Neanderthals were fucked-up because their low effective population size led to inefficient purifying selection. But the usual harmonic-mean effective population size is not quite right: for salvage mutations, things that somewhat correct the wrongness caused by increased genetic load, the effective population size is closer to the average. In other words, there were probably a lot more Neanderthals during interglacial periods, and these genetic consequences of those population booms ameliorated their genetic problems.
Second, Neanderthal Y chromosomes and mtdNA have even smaller effective population sizes, 1/4th as big as the autosomal effective size, and thus were even more likely to be messed up. which is why, maybe the Neanderthals seem to have picked up mtDNA from some African group closer to modern humans around a quarter of a million years ago: they needed it.