You're going to say it's because of physical advantage? Which has been shown to be negligible in trans women who have been on hormones for a length of time, when compared to the variations in physical attributes already present among cis female people (and not even taking into account intersex people who were "corrected " to female at birth). And which doesn't even apply in a number of sports. For instance in shooting and biathlon events being female can be an advantage (at least by averages, there again individuals differ widely within the sexes) so a trans man could theoretically dominate if the hormones don't change that too much. Or maybe we're talking about how compliance to societal standards has always been more strictly enforced for women, even in the definition of being a woman, and how they're perceived as needing protection and separation but/therefore not compensation for their athletic performance.

Try thinking about this: a trans woman who competes as a woman has little to gain because women's sports aren't sufficiently appreciated. If she were willing to deny her identity and compete in men's tourneys she would be up against bigger stronger men, as would any man of her build, and rank lower, but she'd probably have better pay and equipment and facilities regardless. She'd also still probably get similar abuse from transhaters, because she's a target for their misogyny that gets them in less trouble than beating up on "real" women (although they likely do that too).

Trans men are more able to "pass" in society because our expectations about the diverse appearance of men is so much greater. And if a trans man athlete competes and takes a spot from a less-athletic cis man, is society going to support his protest or say he's just a whining wimp? That's an attitude rooted in misogyny as well, by the way. Weak equals female equals unworthy.