I don't think you're wrong here; it's just uncommon to refer to someone as they/them even indirectly while they are present and engaged in the conversation that's happening, but may not be the directed recipient of the statement. In almost all cases, at least until recently, the pronouns he/she or him/her would be used instead; therefore it's sort of awkward to use it as a direct pronoun in that context; but using they/them as a direct singular is not new in any capacity and I believe you're correct on that. It's just uncommon, and IMO, would generally come across as mildly dismissive or insulting toward the individual in question.

I believe the (former) dismissive/insulting nature of the context of referring to someone as they/them directly is the root of the discomfort most people (especially cis-normative persons) have around using the term for direct reference of a singular individual. Their brain is uncomfortable at the fact that they're using (mildly) offensive language towards someone who they likely mean no offense to, meanwhile to use he/him or she/her instead is likely going to be far more offensive to someone who is non-binary, so the discomfort only lies within the speaker and their expectation of how what they are saying will be understood.