A book that I really appreciated was "Racism without racists". It looks at the question of why racism is so prevalent when there aren't many people who consider themselves to be racist? And one answer to that is that many people who aren't actively racist (or at least, don't consider themselves to be, and may not be considered racist by other people) often still unintentionally perpetuate systemic racism, precisely because of the widely held notion that racism happens because of racist people. It's more complicated than that. (This is why it's not enough to be non-racist, we must strive to be anti-racist if we want to combat systemic racism).

In a similar vein, I would say that the hypothetical person you're asking about would be perpetuating transphobia, but I don't necessarily know if I'd call them transphobic — mostly because I think "transphobic" is not a particularly useful term in this context. People tend to become defensive when they are called transphobic/racist, and that can make it harder to convert people from being non-[racist/transphobic] into anti-[racist/transphobic].

I think it's probably better to move to a model of thinking of harms like racism and transphobia as things that you do, rather than things that you are. For instance, although I would consider myself to be both anti-racist and anti-transphobic, there are still times where I think or do things that are racist or transphobic (in the sense of contributing to systemic racism or transphobia). However, the reason why I consider myself anti-racist and anti-transphobic is precisely because I work to hold myself accountable for these things, and to understand my part in the wider systemic oppression (and how I can undermine said oppression).

The transphobia example really sticks out to me, because as someone who is cisgender and bisexual, I always considered myself a trans ally, but it was in a nebulous, ineffectual way that is a big part of why "ally" has some negative connotations these days. I was anxious about doing or saying something wrong, and that meant that considering myself an ally was mostly of benefit to my own sense of being a good person. I knew trans people, but I wasn't super close to any of them.

However, when my long term partner came out to me as trans, that catalysed me learning a heckton in order to support her, which made me realise how useless my "allyship" had been previously. I don't know if I would consider that earlier version of me to be transphobic, but certainly I was unintentionally perpetuating systemic transphobia far more than I do now.

Is it productive to call someone transphobic ever (at least to their face)?

We would guess it does no good to most people as they instantly get defensive because most people care more about not being called a bigot than actually not being one.

Edit: Look around lemmy and reddit etc, it's filled with people like this.