In Chapter 2, Encounters With Neanderthals, David Reich talks about his work in analyzing the first successfully sequenced Neanderthal genomes, and the discoveries that led to.
Reich’s team, with Nick Patterson making an especially important contribution, found that Neanderthals were about equally [genetically] close to Europeans, East Asians, and people from New Guinea, but closer to all non-Africans than to sub-Saharan Africans. This was what you would expect if the ancestors of non-Africans had interbred with Neanderthals, while sub-Saharan Africans had not.
Even thought these results were statistically very strong, Reich was skeptical because it went against the scientific consensus of the time – that there had been no admixture between anatomically modern humans and archaics such as Neanderthals as AMH expanded out of Africa. This was a question that I and John Hawks were interested in. We had concluded that the “scientific consensus” was based on nothing and put no stock in it. We had predicted that such admixture would be found, and that sometimes Neanderthal alleles would have conferred selective advantages and become common. Our reasoning went as follows:
The evidence for zero mixing was weak. It was clear that modern humans did not carry Neanderthal Y-chromosomes or mtDNA, but that could have occurred because they reduced fitness (a slight reduction have been enough to eliminate them, over tens of thousands of years) or because they were never common and were later lost by chance. Y-chromosomes and mtDNA are only two loci. We had checked what was known about successful hybridization in mammals: it turned out that after two species separated, it usually took a couple of million years for serious genetic incompatibility to develop. But we knew, from the fossil record, the Neanderthals split off around a half million years ago. Hybridization should have been possible. Moreover, if it happened, even at a low level, it would be an efficient means of transmitting new favorable alleles to modern humans, some of which might be common today.
One reason [ we think] for that bogus scientific consensus was a confusion of terms. Paleontologists call two sets of fossils separate species if they look sufficiently different [morphological species] while biologists usually define separate species as populations that can’t interbreed. Those aren’t equivalent definitions: we can distinguish the skeletons of Labrador retrievers and poodles, yet labradoodles happen.
Another reason: Ernst Mayr, a prominent figure in biology had opined that hybridization was an unimportant factor in evolution, as far as I can tell for no particular reason at all. For equally mysterious reasons, people paid attention to him.
Anyhow, Reich’s skepticism led to even better evidence for mixing with Neanderthals, which is the way it’s supposed to work in science. By looking at the average length of Neanderthal-derived segments of the genome, they were able to make a rough estimate of the time of mixing. Later work, using ancient DNA from a modern human that lived and died in Siberia about 45,000 years ago, gave a more accurate of 50-60 thousand years ago for Neanderthal admixture. The fact that the amount of Neanderthal admixture was not very different in widely separated populations in Eurasia suggested that admixture occurred in the Middle East, an inevitable first step in expanding out of Africa.
Later work showed something very interesting: the amount of Neanderthal ancestry ( in regions that actually do anything) has been slowly declining over time. In certain regions you see no Neanderthal ancestry at all – Neanderthal gene deserts – bu the trend is general. Not universal, since some Neanderthal genes appear to have been useful and have become common, but the general trend is the gradual shedding of the minority genome.
There are a couple of explanations under consideration for this genetic rejection. One is that Neanderthals were messed up: because their long-term numbers were lower, particularly at glacial maxima, purifying selection would have been less efficient in Neanderthals, and slightly deleterious mutations would have accumulated to higher levels than in anatomically modern humans. We have more example: it looks as if most of the functional part the Denisovan genome is also being slowly rejected by Melanesians. But there’s more: West Africans seem to have have picked up genes from an archaic group in Africa that are slowly being rejected, while Altai Neanderthals seem to have picked some modern human ancestry ( possibly from the Qafzeh-Shkul population in the middle East around 100,000 years ago) and they were rejecting the modern human genome. I can’t see how Neanderthal genomes could simultaneously be both better than and worse than modern human genomes: I think it more looks like a kind of mild incompatibility, where every subspecies is rejecting minority DNA from any exterior source. If true that would be very interesting. But I could be wrong. Probably am.
Reich believes that moderns and Neanderthals were at the edge of biological compatibility, probably with reduced fertility. The evidence is that there is an especially low level of Neanderthal ancestry on the X chromosome, a known hotspot for fertility problems in crosses between related species. And that’s good evidence. But on the other hand, comparisons with other mammalian sister species indicates that it usually takes a considerably longer separation before real fertility problems show up – enough, say, to materially inhibit gene flow.
Probably we need to define terms. I think that a few-percent decrease in fertility in people with large amounts of Neanderthal ancestry might be enough to create those Neanderthal gene deserts on the X chromosome – but at the time, nobody would have noticed it. Smallish effects had something like 2000 generations to play out.