The aliens weren't indigenous to any of the planets the humans found them on, and in Alien 1 we're lead to think they possibly wiped out one ancient civilization. (although they might have only been travellers).

But it def could be the case of colonizing the colonizer, IE the fear that what humans are doing could also be done to them. Much like how aliens force men to give birth against their will.