It only takes a few people to introduce a blob of diversity into an otherwise isolated population and suddenly all their ancestors become contributors to that areas gene pool.
I researched into this, and was incredibly surprised. The only compatible offspring of neanderthals and modern day humans, were with a male homo neanderthalensis, and a female homo sapien. Estimates are you'd only need 1 successful interbreeding every 70 generations or so.
If it's that low, that makes me ponder: How is it we had both enough genetic isolation for speciation to occur, but enough proximity for pairing with modern humans? Denisovans also interspersed among our population, much the same way.
And if it takes that little, why do we see so many distinct genetic populations today? We see distinct differences in bone structure, birth canal shape, height, metabolism, and the article says even specific notable changes to immune system.