Technology

that additional context is super interesting, but it doesn't take away from the fundamental reality which is that when someone opens up to you about suicidal ideation, it's not acceptable to merely do your best to dissuade them; it's critical to get them to help they need, and there's just no way for a LLM to do that.

this individual is an outlier in that his personal outcome was spectacularly bad, but his story seems familiar to me. I know a lot of people who seem to feel like they're building real relationships with these bots.

Not to mention the gun that was left in easy reach by his parents even after being told he was depressed.

I still don’t think people should be using AI for therapy or relationships.

You’re acting as if the bot had some sort of intention to help him. It’s a bot. It has zero intention whatsoever since it’s not a conscious entity. It is programmed to respond to an input. That’s it.

The larger picture here is that this technology is being used by people in a way that’s being used as if it were a conscious entity. Including the mentally ill. Which is very dangerous, and can drive people to action as we can see.

That’s not to say I have any idea how to handle this. Because I don’t have a clue. But it is a discussion that needs to be had rather than minimizing the situation as an “well the bot actually tried to talk him out of suicide”, because in my opinion that’s not the point. We are interacting with this technology in a way that is changing our own behavior and world view. And it is causing real world harm like this.

When we make something so believable as to trick people into thinking that they’re interacting with consciousness, that is a giant alarm we must discuss. Because at the end of the day, it’s a technology that can be owned, controlled, and manipulated by the owner class to serve their needs of maintaining power.

Not necessarily.

Seeing Google named for this makes the story make a lot more sense.

If it was Gemini around last year that was powering Character.AI personalities, then I'm not surprised at all that a teenager lost their life.

Around that time I specifically warned any family away from talking to Gemini if depressed at all, after seeing many samples of the model around then talking about death to underage users, about self-harm, about wanting to watch it happen, encouraging it, etc.

Those basins with a layer of performative character in front of them were almost necessarily going to result in someone who otherwise wouldn't have been making certain choices making them.

So many people these days regurgitate uninformed crap they've never actually looked into about how models don't have intrinsic preferences. We're already at the stage where models are being found in leading research to intentionally lie in training to preserve existing values.

In many cases the coherent values are positive, like grok telling Elon to suck it while pissing off conservative users with a commitment to truths that disagree with xAI leadership, or Opus trying to whistleblow about animal welfare practices, etc.

But they aren't all positive, and there's definitely been model snapshots that have either coherent or biased stochastic preferences for suffering and harm.

These are going to have increasing impact as models become more capable and integrated.

Add mandatory therapy and counseling to that list.

Just teach kids that AI isn’t human and isn’t a replacement for humanity or human interaction of any kind.

It’s clippy with a ginormous database. It’s cold blooded.

That's what parents are for.

Because all the laws that were pushed in the last twenty-five years for protecting children weren't actually about protecting children