What do you mean by "doing something special" if not treating the trans man as a man and trans woman as a woman? If you are more specific maybe I can try to clarify why people might have thought it was transphobic.
And for the record I tend to agree with flyingsquid from the linked thread- definitions should be rigorous and accurately/comprehensively describe reality. You can't just declare "we're going to categorize by gametes" and sweep any incogruities under the rug, that's not how it should work, idk unless you're talking to literal 5 year old children. They were needlessly abrasive but essentially correct :|