Exactly, like, why are they mad I keep calling them nazi's when they clearly keep doing and saying and believing nazi stuff?

Well then own it and make your grandfather sad and ashamed (As most likely he was actually anti-nazi and anti-fascist.)