No I think this genuinely comes from a fear of trans people. Transphobia. But since you're being honest and good faith, and not hateful about it, I can see why it's confusing to be called transphobic.