Dynasties decay. The founder generally has a lot on the ball – tough, a natural leader, and canny campaigner – but his son is unlikely to be so exceptional. Partly this is a manifestation of regression to the mean, and partly because his mother was probably chosen for something other than her talents as a warlord. By the fourth or fifth generation, it can be hard to believe that the useless poet on the throne is truly a member of the Golden Family.
This decay is a fundamental historical fact – an inevitable consequence of biology and primogeniture. It’s one of the important weaknesses of dynastic rule. The Ottomans, however, found a way around it for some centuries – the law of fratricide. Upon the death of the Sultan, all of his sons were theoretically eligible for the succession (not just the oldest). Since the Sultan had a harem, there were a lot of them. Whoever first seized power then had all his brothers and half-brothers executed by ritual strangulation. Incompetents didn’t win out in this struggle.
In practice the law of fratricide was not quite this simple, and imperfectly fair. Older sons were assigned provincial governorships, and a Sultan could assign a favored son to a close-in province, increasing his chance of being the first to Istanbul. This ensured that the successor had administrative experience. It cannot have been pure truncation selection for genetic awesomeness, because younger brothers (toddlers, for example) were at a disadvantage.
This system was effective. The first few Sultans were competent, even before the law of fratricide. Partly that was due to a less formal forms of competition for the throne, involving rebellions and civil wars, but some was probably luck: that’s why the Ottoman state existed rather than an expanded Sultanate of Rum or whatever. The law of fratricide kept the streak going: the Ottoman state had ten competent rulers in a row. I think no other state can make such a claim.
The system worked, but I doubt if anyone enjoyed it. Starting around 1600, things changed. The ‘rule of elderness’ was adopted – all males within an older generations were exhausted before the succession of the eldest male in the next generation. Eligible males other than the new Sultan were locked up in the kafes [‘the cage’], a suite of rooms deep within the Topkapi palace. Their education ceased. They were allowed barren concubines, and some other recreations, mainly macrame. Some of these people emerged much later as the new, deeply confused Sultan: the last Sultan, Mehmet VI, emerged at the age of 56, totally ignorant of the Empire and everything else, other than macrame.