This is a thing i struggle with a lot because i realized i was trans late. So the term makes sense to me on a superficial level. But it also ignores the lifetime of struggle i had trying to fit in as a male, and disliking so many aspects of male socialization. The term is avoiding conflict, while still denying the experience of trans women.
Also to add to your point, so much of anti-trans rhetoric is structured around the idea that all cis men are actively preying women.