The fossil record is sparse. Let me try to explain that. We have at most a few hundred Neanderthal skeletons, most in pretty poor shape. How many Neanderthals ever lived? I think their population varied in size quite a bit – lowest during glacial maxima, probably highest in interglacials. Their degree of genetic diversity suggests an effective population size of ~1000, but that would be dominated by the low points (harmonic average). So let’s say 50,000 on average, over their whole range (Europe, central Asia, the Levant, perhaps more). Say they were around for 300,000 years, with a generation time of 30 years – 10,000 generations, for a total of five hundred million Neanderthals over all time. So one in a million Neanderthals ends up in a museum: one every 20 generations. Low time resolution!
So if anatomically modern humans rapidly wiped out Neanderthals, we probably couldn’t tell. In much the same way, you don’t expect to find the remains of many dinosaurs killed by the Cretaceous meteor impact (at most one millionth of one generation, right?), or of Columbian mammoths killed by a wave of Amerindian hunters. Sometimes invaders leave a bigger footprint: a bunch of cities burning down with no rebuilding tells you something. But even when you know that population A completely replaced population B, it can be hard to prove that just how it happened. After all, population A could have all committed suicide just before B showed up. Stranger things have happened – but not often.