Why should a group defined by plundering travelers be more acceptable than a group defined by being short with green skin?
Because in a fantasy world, where we can know for 100% certainty that gods created life, it's not impossible for those gods to have made a certain creature type objectively evil.
In some settings, Orcs are the way they are because their god is the last one to pick a place for them to live, gets pissy, and decides that "Fuck you guys! If that's how you want to play it, my orcs are going to plunder the shit out of your guys' lands!"
In other settings, there has to be some kind of cosmic balance to things, and some gods are just evil because there has to be a natural counterpart to good, and so the creatures they create are just inherently evil.
I think the issue is with this kind of debate is that that it's referred to as "race". We don't really have a one-for-one on this IRL (because Goblins don't exist) and we don't refer to animals as "different races".