He challenged his critics to a fight.
No he didn't. Saying people are scared to fight for what they believe in does not invite or mean they are afraid to attack him physically. Regardless, someone saying you're scared to fight them, and you actually committing assault to prove them wrong, makes you the idiot in that scenario.