I'm going to assume you're a cis man for this.

Let's say we work together. I insist that I don't care about gender and see no reason why I should. As such, I decide to refer to you as 'lady' or 'girl' over the course of our daily conversations.

Would that not bother you? You don't think that wouldn't eventually start having an effect on you? Being constantly misidentified, misgendered, or having a part of who you are be literally disregarded?