Not really intended to be an insult inherently, but in practice it's similar to the way Jew is a descriptor that can be given a very different meaning based on who says it and how.

You aren't going to say or use the word Jew in the same way David Duke would in the middle of an unhinged antisemitic rant. I'm not going to say or use the word Goyim in the same way as an Israeli settler whose hatred has similarly divorced him from reality.

Hope this makes sense.