Let's leave aside the labels (sexist, racist, etc) for a moment, because these conversations tend toward applying/avoiding those and it just loses a lot of nuance.

Let's metaphor this, because I think that helps. Is it possible for someone with millions of dollars to have a truly bad day? Of course it is. Is it possible for them to be hurt by someone with way less money than them? Obviously, yes. Positions of privilege never fully insulate anyone from hurt or harm, and those in worse positions can perpetuate harm. That's fully understood and accepted.

I don't think anyone with integrity would say that women are in a position of power relative to men. Women have been systemically and systematically oppressed for virtually all of human history. A woman even being able to talk back to a man without severe physical consequences is an insanely recent development at scale in our world. There are still dozens of countries that are not letting women wear what they choose, marry who they choose, go to school. Men (as a group) have never been subjected to anything remotely close to anything like this, and in fact have perpetuated it for all time.

Now, there are some whackos out there who hate all men because of that. They're super, super rare, and they're wrong. Most women are indeed wary about random men, especially if they have experienced assault or harassment, but that is a far cry from hating all men.

To boil it down, there's a huge historical and modern difference in the way the genders/sexes are treated, and that cannot be ignored just so we can try to achieve the utopian world of no distinction. We have work to do as a society, as genders, and as individuals to repair this gap together. Good men belong right next to us, doing that work. And every good man I've ever met has willingly done so. Instead of asking "why are you avoiding me?", they give us space and support. Instead of asking "why not men?", they do the work to support fellow men instead of asking women to do it for them. Instead of saying "not all men", they actively engage in not being those men and are content in that.