I think a better explaination is simple in-group out-group behaviors. Humans have a tendency to separate themselves into in-groups (us) and out-groups (them). And the most obvious differentiation between people is the way they look. So peoples brains made a consensus that "people who look like this are the in group, and everyone else is the out group". And the in-group gets trust and preferential treatment, while the out-group is feared and excluded. Once the groups are defined, explainations about intelligence or aggression or moral character are applied post-hoc to explain why the outgroup is excluded