I've given a lot of thought to this issue in the past and I think it all boils down to one indisputable fact:

"You just believe her" is completely at odds with "innocent until proven guilty."

"We should believe women" is a laudable phrase, and it makes us feel good to say it, but men are victims as well, especially trans men. "We should believe victims" would be better, but it is a begging-the-question fallacy, it assumes the victimhood is true. The people who made that not possible are specifically the people who have made false allegations in the past.