It didn’t all happen in a day. It took years to build up the the worst of it.

That's literally the point.

You deal with nascent/rising fascism through different means than when it is full blown. Public opinion still matters at this stage, unlike when fascism has taken hold.

and it would give them the PR that they want.

You even seem to understand that the violence is exactly what they want.

should have been run out of town at the very least.

Doxxed, fired, jeered etc. Yes, nazi clad fucks shouldn't be comfortable. But once you throw that punch, you've lost. And the ones wearing the nazi regalia want you to throw that punch. It's the same thing as brawling with ICE, in both cases you are acting exactly how the fascist wants, and arguably requires, you to act.

By throwing those punches, or encouraging others to, you are helping the fascists win.