I am often in workwear (big canvas jackets, steal toed boots, ill-fitting pants) in public bathrooms because of my job. I'm also a retired athlete who is built a little bigger than the average woman. I have been screamed at by a woman in home depot saying, "You're not supposed to be in here. This is the ladies." There have been a few other times I have gotten glares and suspicious looks trying to figure out what I have going on under my large clothes.
I have a theory that most of this trans bathroom fear has resulted in negativity towards other women. Butch women exist.