I'm not invalidating your experiences, and you do you. I wouldn't put it past shitty people to use it passive-aggressively as a way to misgender someone, and if you've experienced that, I am truly sorry you've had to deal with such shitheads.
At the same time, I know plenty of trans and cis women that don't see it as rude or invalidating of their gender identity, and even use it themselves to refer to groups of people with mixed genders.
It's a matter of boundaries and knowing one's company. Some people are cool with it, and some people aren't. It would be nice if everyone was using ungendered terms by default, but that's going to take a while, unfortunately.