If I'm not trying to get in your pants, your sex shouldn't matter.
And I don't know why I would treat women differently then men, so I'm not sure why I need to take gender into account either.
It's different if were discussing things that are gender/sex exclusive, like periods or something.(I can't offer gender exclusive issues bc i cant think of any that arent just gender roles, which i despise) But for casual conversation why does gender matter?
To be clear, I am not trying to convert you here, I just figure you can give me a better explanation if you understand where im coming from.