Ultimately, it's all a wash. The nazis we are punching would say their targets broke the social contract first. Tolerance should be an aspect of your morality, not the whole thing (and hopefully, that being said, your morality says punching nazis is okay).