Victims and perpetrators aren't 1-1. Sexual violence isn't distributed normally - minority of men commit a majority of the sexual assault.

Knowledge of that perpetrator is spread through the network of every woman they victimize, while it is limited to his own personal network of male non-victims.

This is before considering the fact that people are more likely to discuss being a victim than being a rapist. Even if each rapist had exactly 1 victim, it would still be more likely for women to know someone who was raped than for the male acquaintances of the perpetrator to know that they know a rapist. Doubly so given that many rapes are not officially reported, and many that are do not get prosecuted.

I have many uncles and male cousins. I don't think any of them are rapists, but the stats are pretty decent at least one is. All of my aunts have stories of perverse behavior by males, and one was kidnapped and raped repeatedly.