755
Father sues Google, claiming Gemini chatbot drove son into fatal delusion (techcrunch.com)
submitted 7 months ago by to c/technology@lemmy.world
233 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 34 points 7 months ago* (last edited 7 months ago) (10 children)

That's fucking crazy. Did he ask it to be GM in a roleplaying choose-your-own-adventure game that got out of hand, and while they both gradually forgot that it was a game the lines between fantasy and reality became blurred by the day? Or did it just come up with this stuff out of nowhere?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 10 child comments
    •
  • [–] 62 points 7 months ago* (3 children)

    In every other case of AI bots doing this, the bot will always affirm whatever the person says to it. So if they say something a little weird, the AI will confirm it and feed it further. This happens every time. The bots are pretty much designed to keep talking to the person, so they're essentially sycophantic by design.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 11 points 7 months ago (1 child)

    I just tried this with ChatGPT three days ago and there’s a chance they have tried to make it slightly less sycophantic

    I was essentially trying to get it to tell me I was the smartest baby born in whatever year like that YouTuber—different example but it was so resistant to agreeing to me or my idea or whatever being unique/exceptional.

    Hope this is a specific direction and not random chance, A/B testing, etc.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 10 points 7 months ago (3 children)

    That would be my bet, LLMs really gravitate towards playing along and continuing whatever's already written. And Gemini especially has a 1M long context so it could be going back for a book's worth of text and reinforcing it up the wazoo.

    That said, there is something really unhinged about Google's Gemma series even in short conversations and I see the big version is no better. Something's not quite right with their RLHF dataset.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 7 months ago

    I have found Gemini the hardest to jailbreak tbh. I have been able to get Claude and CGPT to straight up give me a list of curses and slurs it isn't allowed to say, but Gemini will only do it if you say the words first.

  • permalink
  • fedilink
  • source
  • parent