I'm going to over a possible counter view, with the caveat that I don't know your specific situation.
I'm not a big guy, in fact, I don't see myself as a guy at all. Certainly I don't think of myself as intimidating in any way. And yet, I've come to learn over the years that when I get upset/angry, people absolutely do find it threatening. It's a hard thing to understand because it doesn't match my self image at all, and it's not something I can control. I feel like, "Who could possibly be intimidated by me?? I'm just a little nonbinary entity!" But for whatever reason, they do, and I'm in the process of learning how to adjust my expectations accordingly.