There was study that determined that the remnant neanderthal DNA in humans mostly came from neanderthal male and human female pairings and not the other way around.

As others have pointed out in the comments that's not really proving that the other pairings didn't happen as often or even more often, there could be fertility issues with the opposite pairing and a whole host of other factors.

It's enough to make a silly meme tho