There is still doubt, but there seems to be a good chance that the Flores Hobbit was a member of a distinct hominid species, rather than some homo sap with a nasty case of microcephalic dwarfism. If this is the case, the Hobbits are likely descended from a small, Australopithecus-like population that managed to move from Africa to Indonesia without leaving any fossils in between, or from some ancient hominid (perhaps homo erectus) that managed to strand themselves on Flores and then shrank, as many large animals do when isolated on islands.
Island dwarfing of a homo erectus population is the dominant idea right now. However, many proponents are really bothered by how small the Hobbit’s brain was. At 400 cc, it was downright teeny, about the size of a chimpanzee’s brain. Most researchers seem to think that hominid brains naturally increase in size with time. They also suspect that anyone with a brain this small couldn’t be called sentient – and the idea of natural selection driving a population from sentience to nonsentience bothers them.
They should get over it. Hominid brain volume has increased pretty rapidly over the past few million years, but the increase hasn’t been monotonic. It’s decreased about 10% over the past 25,000 years. Moreover, we know of examples where natural selection has caused drastic decreases in organismal complexity – for example, canine venereal sarcoma, which today is an infectious cancer, but was once a dog.
There is a mechanism that might explain what happened on Flores – partial mutational meltdown. Classic mutational meltdown occurs when a population is too small for too long. Selection is inefficient in such a small population: alleles that decrease fitness by less than 1/N drift fairly freely, and can go to fixation. At the same time, favorable mutations, which are very rare, almost never occur. In such a situation, mutational load accumulates – likely further reducing population size – and the population spirals down into extinction. Since small population size and high genetic load increase vulnerability to disaster, some kind of environmental catastrophe usually nails such doomed, shrinking populations before they manage to die off from purely genetic causes.
In principle, if the population is the right size and one adaptive function is considerably more complicated than others, presenting a bigger mutational target, you might see a population suffer a drastic decline in that function while continuing to exist. There is reason to think that intelligence is the most complex adaptation in hominids. More than half of all genes are expressed in the brain, and it seems that a given degree of inbreeding depression – say cousin marriage – depressesIQ more than other traits.
Flores is not that big an island and the population density of homo-erectus type hunter-gatherers must have been low – certainly lower than that of contemporary hunter-gatherers, who have much more sophisticated tools. Thus the hobbit population was likely small. It may not have been possible to sustain a high-performing brain over the long haul in that situation. Given that their brains performed poorly – while the metabolic costs were as high as ever – selection would have shrunk their brains. Over hundreds of thousands of years, this could well have generated the chimp-sized brain we see in the LB1 skeleton.
Of course, this could only have happened if there was an available ecological niche that did not require human-level intelligence. And there was such an opening: Flores had no monkeys.