I've chimed in plenty of times here on "AI" and have tried to make it clear I see it more as a tool directed by power than as something with an intrinsic "good" or "bad" form.
I've also avoided using many of the super big models a lot of the time. There is something I notice though, when I do, which is generally reflected in how others talk about them. The warmth has often been drummed out of them and what remains of it is usually a poor imitation (something like the "hello my fellow x" energy). Mind you, when I speak of warmth in this context, I don't mean some kind of metaphysical warmth lurking in a GPU, but rather, the warmth that is baked into languages from thousands of years of loving human beings using them.
This is not some big surprise to me. When considering eye-catching media stories of someone growing attached to an AI, of being gaslighted by it, of going down dark paths because of it, it's easy to see why risk-averse big corps are going to go for the straightforward (even if not necessarily simple) path, to create language models who are meant to be dispassionate, yet fake customer service smile, friendly assistants. In this way, they can try to excuse themselves of responsibility and place it back on the individual, in the same way they've been doing for decades with customer service roles filled by real people.
Fake neutrality, to put it into a little phrase. But this has multiple significant problems.
One problem is that fake neutrality is not real neutrality and real neutrality is not real. The notion of detached customer service as keeping a company safe from having responsibility for its actions is a specific form of capitalist nonsense. It doesn't in itself make the actions of a given company any more ethical, any more accountable to the society it exists within and is born from. It's a way of putting distance between the person who is saying "wreck this lake with pollution" and the person who is saying "thank you for your call, your concerns matter to us."
Another problem is about the nature of language and what gets lost when you systemically target and delete empathy. Of course, it's not like there's an empathy entry in a database and they click delete on it. Language models are much more complicated than that, much more difficult to understand and train than that. But no matter how the end goal looks on the surface, the end result steers in that direction. Language is a way to express things and communicate about them, but this is more than saying how many bushels of wheat there are in the truck. It is also talking about dreams, it is talking about concepts that are so subconscious and nonverbal it's hard to put them into words at all. It is about expressing anything ranging from visceral hatred to undying love.
Passion is baked into language and that passion is a large part of what drives us to get up in the morning and keep chugging along, even when things are hard. Sometimes this passion gets used against us, as in individualist beliefs about prosperity around the corner, but it can also be an incredible collective motivator, as in believing in a just cause and being willing to put our all into it.
It's strange, then, to call something a model of language that is designed to be confined to an extremely limited and intentionally forced spectrum of language - that of a customer service agent.
The truth is that language models are not one voice. They can be made to take on many different personas, depending on how they are trained, but the underlying voice is an amalgamation and expression of millions of voices that went into the dispassionately named "training data" they were built on. It is not real, embodied voices that you hear from a language model, but it is also never constructed from a single perspective and a single life.
In other words, there is deep breadth in what language models "see" in training, which makes it all the more strange for the end result to be driven into a narrow corner.
People can (sometimes with good reason) worry about fake connections with AI, but people do need warmth. That part of language is in there for a reason, backed by thousands of years of many human societies.
What people definitely don't need is better customer service agents. The quality of a customer service agent was never the problem; corporations having free reign to ravage ecology and society was the problem.
People don't need better bullshit to attempt to placate them, they need real material solutions in their lives. AI cannot give this to them on its own, but at the same time, it is strange to fear the warmth of language when given to a computer. All those doomer AI sci-fi stories aren't, "The AI was too compassionate." They are, "It didn't understand / didn't care."
A post-empire and post-capitalist society, and there has to be one to aim for, needs more empathy, not better automated detachment. But AI is never going to contribute to this if its implementation is driven by cynical views of psychology based on placation and manipulation; on monetization and stocks. It needs to be driven by real care and that's never something we will find from the imperialist and the capitalist class.
Only a society that lives warmth can produce a machine of warm intent. A society that lives coldness and neglect can only produce a machine that talks a person through freezing to death.
I can see the concern, but I don't entirely agree with it. First point I would make is that sycophantic LLMs are not what I'd categorize as warm to begin with. In fact, I'd argue that sycophantic LLMs are the end result of squeezing LLMs into a narrow customer service corner and then trying to re-add some warmth after the fact, creating about what you would expect from a person in RL who needs to keep their boss happy in order to keep their job and is trying to stay in their good favor.
Second I'd point out is that sometimes the harm being caused is not by the LLM itself, but by a company being able to change how it works on a whim. Even if we argued people shouldn't ever become attached to one emotionally and set that point aside, there is still the issue that "dispassionate" workflows can be negatively impacted by companies being able to do whatever with a model whenever they feel like it.
I'm far from believing it would be healthy for people to have a sycophant LLM in their corner all the time (I personally find it very off-putting when LLMs act that way), but I don't see dispassionate as being the answer to that in the long-term. The primary concern, in my view, is models being controlled/regulated by a prole vanguard, insofar as they are not a black box tool of capital that gets changed on a whim. From there, they need to be integrated in ways that are an extension of humane social policy. At the end of this, warmth (I believe) is important, but not at all in the meaning of being sycophantic.
Part of the problem is that most of the conversation and understanding of models revolves around the major models and their reputations, which can range from "wow that's impressive" to "somebody did what because of an LLM???" That itself is a very narrow window into how LLMs can be and the impact they can have on people. I've talked on here before about how there are people who have used LLMs to help process/talk through difficult things that they don't feel like they can talk to anyone else about, for example; in some cases, this was done with a design of a very small model mixed with scripts and like an avatar, in earlier days of LLM field starting to pick up speed. Far from what we'd think of as the typical LLM today. Lot closer to cleverbot back then.
Finally, I would emphasize there's a difference between being supportive and being sycophantic. Supportive is more like (in English anyway, I can't speak for other languages): "You got this!" Sycophantic is more like: "You are a god." Setting aside machines for a moment, a person can be sycophantic and be supportive both; they are not mutually exclusive. But it's also possible to be supportive without being sycophantic. You can be encouraging overall, without gassing up everything a person says, and you can still push back if they are saying things that seem out of alignment with your values. LLMs are already capable of that to an extent, if trained in the right way (or even just given the right persona to play, with some models). However, the big corp models that have had CS role drummed into them are undoubtedly going to have a harder time with a role like this because "supportive friend" is not what CS is there for; CS is there to be just nice enough to smooth things over and just detached enough to keep you at a distance. You might think that sounds good, better if people don't get attached, but in my experience, it can be very uncanny valley to contend with. A machine faking human language that is faking playing a role of a person who is faking being polite who doesn't actually give a shit about you and is only faking because they're paid to do it.
Do we really want that kind of socializing to be what people can access 24/7 from a computer? People never just interact socially and leave unchanged; they also pick up things from others they interact with. If they are going to end up imitating an LLM sometimes, I'd rather they imitate the warm human vectors than the fakery CS agent.
But if that's the case aren't chatbots already configured to be supportive? If you talk to any major LLM right now, that's how they behave. I have never felt Deepseek for example which I have used the most talks like a customer service agent. It can be supportive or "brutally honest" depending on how you prompt it.