Previously they would have had to encounter a person who wanted to manipulate them. Now there's a widely marketed technology that will reliably chew these vulnerable people up.

That has always been the case. Look at any angle Trump voter.

There’s probably already an underlying mental health issue, and it’s just getting exacerbated by the LLM.

I still use the machine that ruined my life and drove me crazy, but only because I’m too lazy to type “lasagna recipe” in to Google.

lmao "core rules that cannot be overwritten" that not how llms work

EDIT: oh, yeah you said the same thing

What's weird to me is they now recognize AI will lie to you but somehow think they can prompt it not to? Your rules can be "overwritten" because they do not exist to ChatGPT. It does not know what words mean.

I can fix her...

There are no more philosophical discussions.

Yeah... if you can't have a philosophical discussion with someone (or something) that's giving you false information or using invalid logical structures, without falling for their bullshit by uncritically accepting everything they say, then you're not having philosophical discussions right, and that's on you...

Agreed, but I think it's also common for people to anthropomorphise these things and common for these chatbots to reinforce and support their users views. I think that's a problem for more people than just those struggling through disorders or an emotionally turbulent time. But I think those people are particularly vulnerable to the flaws, even with functioning mental health and a strong support network. But yeah, a lot of these pieces dramatise and anthropomorphise in ways that aren't necessarily helpful

mental healthcare field being practically non-existent in most countries

I’m in one of those countries so I’m having a hard time imagining how good mental healthcare could intervene. Could you give me an example?

This is one of the reasons I heard one sex doll vendor say their demographics are divorced men over 40 and users want AI in them.

This is like any matchmaking app genuinely attempting to match you with "the one" through AI, algorithms, science, etc so when you meet the perfect person you stop giving the app money.

I got lucky and married my fuck buddy that I met on Tinder. But that is not a good business plan. Why would OpenAI drive people to stop using their product.

I'm a functional alcoholic. Last I checked booze companies aren't reaching out to me to stop buying booze because they care about my personal health or mental wellbeing....

Buuuuulshit

I mean, what are the odds that the statement was composed by an AI?

The billionaires are the cancer. AI is just the newest tool for humanity's self-destruction

Get rid of capitalism and it is fine..

If you pay for them via Openrouter or something then you’ve got an enormous window to work with. Gets more and more expensive as the history increases though.

when did you last use chatbox?

even the last of the pack mistral has memories

If it were a drug, it would be banned by now.

PEBKAC

I learned it as "PEBKAC". Problem exists between keyboard and chair. PICNIC is nice too though.

That's because it isn't true. Retraining models is expensive with a capital E, so companies only train a new model once or twice a year. The process of 'fine-tuning' a model is less expensive, but the cost is still prohibitive enough that it does not make sense to fine-tune on every single conversation. Any 'memory' or 'learning' that people perceive in LLMs is just smoke and mirrors. Typically, it looks something like this:

-You have a conversation with a model.

-Your conversation is saved into a database with all of the other conversations you've had. Often, an LLM will be used to 'summarize' your conversation before it's stored, causing some details and context to be lost.

-You come back and have a new conversation with the same model. The model no longer remembers your past conversations, so each time you prompt it, it searches through that database for relevant snippets from past (summarized) conversations to give the illusion of memory.

I've experimented with chatbots to see their capabilities to develop small bits and pieces of code, and every friggin time, the first thing I have to say is "shut up, keep to yourself, I want short, to the point replies"because the complimenting is so "whose a good boy!!!!" annoying.

People don't talk like these chatbots do, their training data that was stolen from humanity definitely doesn't contain that, that is "behavior" included by the providers to try and make sure that people get as hooked as possible

Gotta make back those billions of investments on a dead end technology somehow

go take a look at https://www.reddit.com/r/EscapingPrisonPlanet/. The venn diagram is a circle.

What kind of person do you have to be to become addicted like them?

Human cognition degrades with stress, exhaustion, and trauma. If you're in a position where turning to an AI for relationship advice seems like a good idea, you're probably already suffering from one or more of the above.

Also doesn't help that AIs are sycophantic precisely because sycophancy is addictive. This isn't a "type of person" so much as a "tool engineered towards chronic use". It's like asking "What kind of person regularly smokes crack?"

Do you need to be very empathetic towards objects? Like seeing faces in everything and get emotionally attached?

I'll give you a personal example. I have a friend who is currently pregnant and going through a bad breakup with her baby-daddy. She's a trial lawyer by trade - very smart, very motivated, very well-to-do, but also horribly overworked, living by herself, and suffering from all the biochemical consequences of turning a single celled organism into a human being.

As a result of some poorly conceived remarks, she's alienated herself from a number of close friends to the point where we doubt there's going to be a baby shower. Part of the impulse to say these things came from her own drama. But part if it came from her discovering ChatGPT as a tool to analyze other people's statements. This has created a vicious behavioral spiral, during which she says something regrettable and gets a regrettable response in turn. She plugs the conversation into ChatGPT, because she has nobody else to talk to. And ChatGPT feeds her some self-affirming bullshit that inflates her ego far enough to say another stupid thing.

To complicate matters, her baby daddy is also using ChatGPT to analyze her conversations. And he's decided she's cheated on him, the baby isn't his, and she's plotting to scam him.

So now you've got two people - already stressed and exhausted - getting fed a series of toxic delusions by a machine that is constantly reaffirming in the way none of your friends or family are. It's compounding your misery, which drives anxiety and sends you back to the machine that offers temporary relief. But the advice from the machine yields more misery down the line, raising your anxiety, and sending you back to the machine.

What's producing this feedback loop? You could argue it is the individual, foolish enough to engage with the machine to begin with. But that's far more circumstantial than personality driven. If my friend didn't have a cell phone, she wouldn't be reaching for ChatGPT. If she wasn't pregnant, she wouldn't be so stressed and anxious. If she wasn't in a fight with her boyfriend, she wouldn't be feeding conversations into the prompt engine.

AI can [...] be completely wrong. But that happens maybe 10% of the time.

Where are you pulling your numbers from, mate? The figures I've seen so far start somewhere >40% and go all the way up to 70%.