We now know that the fraction of Neanderthal ancestry in coding regions has been gradually decreasing with time since the origin admixture, and is now something like half as large as it was originally. There were some useful Neanderthal alleles that were favored by selection, and others that deleterious enough to have disappeared completely, but we’re talking about the general trend.
A couple of papers argue that Neanderthals had a lot of small-effect genetic load resulting from inefficient purifying selection in a small population. They conclude that Neanderthals must have been pretty screwed-up – tens of percent fitness drop – but that can’t really be the case, since they resisted replacement by modern humans for something like 60,000 years after AMH first left Africa.
Moreover, it now seems that the Altai Neanderthals picked up some AMH ancestry roughly 100,000 years ago, possibly from the Qafzeh-Shkul population – and there too, the fraction of minority ancestry seems to have decreased in coding regions. So this isn’t caused by low population size, at least primarily: it’s a kind of incompatibility.
I’m thinking of it as standards drift. In a populations, alleles are always being selected for compatibility, for working correctly, conferring high fitness, on a particular average genetic background. Each allele has a spec it needs to meet. That spec doesn’t necessarily stay the same over time: obviously changes in environment will make a difference. Drift should matter too: if a given allele becomes more common, even by chance, the specs will change for other alleles that interact with it. But there’s always a spec.
When two populations split, their specs start to drift apart. There’s no genetic equivalent of that iridium meter bar. Function at the organismal level doesn’t change so much, but there are many slightly different ways of achieving that function.
So: we now know that if two hominid populations have been separated for 600,000 years, their alleles are on average no longer wholly interchangeable. Admixture and adaptive introgression are still possible, but the alien genome is, in general, slowly rejected.
This means that we can expect the same effect in other cases. Melanesians should have been slowly purging their Denisovan ancestry in coding regions – even though the Denisovans apparently had a significantly higher effective population size than Neanderthals. Since the AMH/Neanderhal/Denisovan split is almost a trichotomy, Neanderthals were probably rejecting Denisovan admixture, while Denisovans were slowly rejecting Neanderthal admixture.
While we’re at it, if there are Pygmies whose genomes are majority ancient Pygmy, their Bantu component is probably slightly incompatible: if left to themselves for a hundred thousand years, they’d probably lose a fair amount of it. Of course they will all be eaten long before that happens.
This is a subtle effect, one that takes thousands of generations to play out. I don’t know if anyone has seen it in the lab – wouldn’t think so.