Absolutely, I have to keep this in mind every time these people say something about "foreigners" (ie. coloured people; to get more specific, [insert slur here]), taxes, or people who get welfare benefits. It's not to say they get a pass on being assholes, but I understand where it comes from.
What really bothers me is the constant use of slurs as if it's the normal way to talk about people. Contrast that to "normal people," ie. whites who are ostensibly citizens.
Sometimes I wonder how I, a white immigrant, am spoken about when I am not present. But then again I don't think I want to know.