Suppose that the narrow-sense heritability of IQ is 0.7 [ in typical western circumstances: no ball-peen hammers), and the non-genetic variation is almost all caused by mysterious unshared-environment effects – not the school you go to or the books in the house, but something essentially random, like somatic mutation, or randomness in development.
Then while a big fraction of variance in IQ is caused by genetic differences, quite a bit is not.
But now look at the difference between two groups. It’s entirely possible that those random forces – somatic mutation, noise in development, etc – are close to the same in both groups.
If so, the difference in the averages of the two groups would be almost entirely genetic, since the non-genetic factors would average out.