The problem is that people keep saying “Not all men” in direct response to women expressing that they don’t feel safe. Same energy as telling a black person “All lives matter”.

I don't think this is entirely accurate. The equivalent of an "all lives matter" person is a "men have issues too" person, not a "not all men" person. The first two are from people taking issue with advocacy for one group (black people; women) because they see it as exclusionary of or harmful to their group (white people; men)*. In contrast, "not all men" people are responding to a percieved attack directed at them, not just to being excluded from targeted advocacy. Subtly but meaningfully different, imo.

* Technically non-black people and non-women, I suppose? But this is the general pattern I've recognized.