I think they think "Nazi" or "anti-semite" is just like "bitch"; it's an insult largely divorced from its linguistic origins. So a person is a Nazi if you don't like what they're doing, and the word means nothing more.

Edit: I think people misunderstood and thought I said these guys aren't Naizs. They are. They just don't like when people call them that because they don't know how words work.