There's one more that I can think of.

Their recent ancestors also migrated there, and look how they treated those who were already there before them. Now, every time they look in the mirror there is a guilt they will not fully acknowledge, but it's there, and deep down they're afraid they'll end up faring like the natives, that they'll end up getting treated the way they have treated others.

That they'll get treated the way they know deep down that they deserve.
No coincidence that those most loudly and vulgarly opposed to immigration are concentrated in the racist, confederacy-sympathizin' pro-slavery south.

It's so easy for charlatans to successfully pop-up in that environment and whip the people into a frenzy, feeding them more ignorance and fear every day. You get the evangelical preachers and the republican politicians, and the line is blurry between the two.