I used to think gendered bathrooms were silly, but my Sex & Gender & American Imperialism professor mentioned that getting separate bathrooms was an early feminist issue, which might explain (a small percent of) the TERF backlash, they see it as "men invading women's spaces" which they fought for. There's also something to be said for rape culture, our current cultural climate might make it actively dangerous for women, cis or trans, to be in compromised positions in spaces men are sanctioned to appear in. Not that men are dangerous, but that dangerous men exist and are shielded by rape culture.