Yeah apparently even Eliza messed up with people back in the day and that's not even an LLM.

LLMs are trained on human writing, so they'll always be fundamentally anthropomorphic. you could fine-tune them to sound more clinical, but it's likely to make them worse at reasoning and planning.

for example, I notice GPT5 uses "I" a lot, especially saying things like "I need to make a choice" or "my suspicion is." I think that's actually a side effect of the RL training they've done to make it more agentic. having some concept of self is necessary when navigating an environment.

philosophical zombies are no longer a thought experiment.

People have this issue with video game characters who don't even pretend to have intelligence. This could only go wrong.

Personally, I hate the idea of not doing something because there's idiots out there who will fuck themselves up on it. The current gen of AI might be a waste of resources and the whole concept of the goal of AI might be incompatible with society's existence; those are good reasons to at least be cautious about AI.

I don't think people wanting to have relationships with an AI is a good reason to stop it, especially considering that it might even be a good option for some people who would otherwise just have no one or maybe too many cats for them to care for. Consider the creepy stalker type that thinks liking someone or something gives them ownership over that person or thing. Better for them to be obsessed with an LLM they can't hurt than a real person they might (or will make uncomfortable even of they end up being harmless overall).