Madagascar was originally settled by Indonesians, specifically people from South Borneo, possibly as early as 300 BC. Recent genetic work on their mtDNA suggests that the number of founding mothers was small, around 30, and those researchers have speculated that the colonization event may have been a shipwreck. That early population accounts for about half of the ancestry of islanders today, the rest mainly being Bantu.
Those castaways had the island to themselves for something like a thousand years: that was plenty long enough to fill it up. Imagine that they averaged three surviving children per couple, which doesn’t sound at all hard in an island with few local diseases and plenty of tasty animals with no adaptation to humans: with a founding population of 60, you would end up with hundreds of millions of people in 40 generations. Impossibly large, but that makes the point.
The number of Africans who immigrated to the islands (much later) must have been far, far larger than that one shipload, but they only end up accounting for half the genes in Madagascar today. Through the miracle of exponential growth, a tiny early colonization has as much effect as large numbers arriving later. In much the same way, there are more descendents of the Puritans in the Massachusetts Bay Colony in this country than there are Italians, even though vastly more Italians immigrated.
This also fits with the story of the three migrations found the Amerindian population. The first migration, although small, accounts for the vast majority of Amerindian ancestry, and the later two (Na-Dene and Eskimo) succeeded by settling land in the far North that was effectively uninhabitable, unless you had the right technology.