Because attacking someone for their looks is criticising them for not adhering to your particular standards of attractiveness. Dismissing and attacking someone for the way is constantly used to put down queer people, fat people, disabled people, alternative people etc. These standards come from misogynistic and racist ideas of what attractive people look like, that is skinny people with white skin and white features.

When you attack someone for looking a certain way, you attack anyone who looks like them. People deserved to be judged for their actions and beliefs, not their presentation. Attacking someone's appearance enforces the idea that your appearance needs to be pleasing to others, and if it isnt, you deserve to be degraded, which is extremely harmful to so many people, especially queer and trans people.