I was looking at a recent survey of current knowledge in psychological genetics. The gist is that common variants – which can’t have decreased fitness much in the average past, since they’re common – are the main story in the genetic architecture of intelligence. Genetic load doesn’t seem very important, except at the low end. Big-effect deleterious mutations can certainly leave you retarded, but moderate differences in the number of slightly-deleterious mutations don’t have any observable effect – except possibly in the extremely intelligent, but that’s uncertain at this point. Not what I expected, but that’s how things look right now. It would seem that brain development is robust to small tweaks, although there must be some limit. The results with older fathers apparently fit this pattern: they have more kids with something seriously wrong, but although there should be extra mild mutations in their kids as well as the occasional serious one, the kids without obvious serious problems don’t have depressed IQ.
Selection still acts on really nasty mutations affecting the brain, even today: people that are crazy or retarded have low fitness. We’re better at preventing early deaths from infectious disease than we used to be, but they were never super-related to IQ in the first place. Selection can change the frequency of those common variants, so differential reproduction with IQ could have happened and had consequences, probably bad ones. But how bad? I doubt if within-population selection decreased IQ by as much as a point a generation. That makes it hard to see how IQ could have declined by a standard deviation (15 points) since the Victorian era – there is no available genetic mechanism. Since it is also obvious (to anyone who has eyes) that no such change occurred, the lack of a feasible mechanism is not really a problem.