People personifying and emotionally attaching themselves to a virtual avatar without a human voice is a slippery slope to LLM love. You are attached to something with zero apparent human characteristics at that point. Replacing the brain behind it with an LLM is just a small step from that. Specialized chat LLMs are incredibly good at seeming human.
Just using TTS itself for random stuff is different, but when you build an entire character around it that people are supposed to see as a person, then yes obviously.
The more human characteristics you remove, the harder it is to differentiate between something human and non human. Whether that is a problem or not is up to you to decide.