alt textCrudely drawn drawing of a person telling a computer "say 'i am in pain'". The computer replies with "> I AM IN PAIN". The person then says "oh my god."

yes, it's a real thing

Crudely drawn drawing of a person telling a computer "say 'i am in pain'". The computer replies with "> I AM IN PAIN". The person then says "oh my god."
you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 3 days ago (51 children)

None of you guys have read the original paper have you?

  • source
  • hideshow 51 child comments
  • [–] 22 points 3 days ago (48 children)

    yea activation steering is not prompting but all things considered the meme still captures the absurdity of believing a program is in pain merely because it outputs human like language mimicking what one in pain says and thinks

  • source
  • parent
  • hideshow 48 child comments
  • [–] 6 points 3 days ago (45 children)

    Exactly. I own a few books that talk about pain, but not for a second do I believe the book itself is experiencing pain. It is simply text.

  • source
  • parent
  • hideshow 45 child comments
  • [–] 3 points 3 days ago (5 children)

    Inbefore anyone says 'but the book doesn't produce text', true, but the printing press did. I see very little difference between llms and printing presses: they both reproduce the words of others, the main difference is speed and scale. The pain you think you recognize in the output of an llm is real: it's real pain experienced by real people, probably hundreds or maybe thousands of people writing about their pain, mashed up and reprinted to suit you.

  • source
  • parent
  • hideshow 5 child comments
  • [–] 2 points 3 days ago (3 children)

    A printing press is direct input to output, it's not having to compute anything nor does it have any internal state that could hold an emotion

  • source
  • parent
  • hideshow 3 child comments
  • [–] 4 points 3 days ago (2 children)

    A model has no internal state either, that's the thing I've been trying to get across this whole time. It is completely static: the model you send your first message to is exactly the same as the one you send you second message and third message, it does not react or change as a response to prompts.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 3 days ago (1 child)

    A models has internal state in each forward pass and the KV cache it builds on reading input is also state.

  • source
  • parent
  • hideshow 1 child comment
  • Whether the input prompt is compressed or not, or whatever form it takes, it doesn't make what I said any less true: the model is stateless. It would be impractical to serve an llm any other way: loading and unloading weights from/to the gpu memory is very expensive in terms of time and power. Cloud llms only makes sense if they can serve thousands of users without changing.

  • source
  • parent
  • load more comments (39 replies)
  • [–] 0 points 3 days ago (1 child)

    You're assuming "mimicking"

    In the study the models repeatedly preferred to press a button to avoid the pain vector even when given no indication what it would do and even when it would cause harm to a user

  • source
  • parent
  • hideshow 1 child comment
  • [–] 3 points 2 days ago

    You’re assuming “mimicking”

    Because it'd be really bizarre if it weren't. We're a Type 0 civilization. We can barely leave our planet. However, we've created sentient and sapient life. Accidentally. On a laptop. And this life we've created, younger than a two year old, can speak human language perfectly. That's some Weird Science level weird. Like creating Kelly LeBrock with an Apple IIe. Thinking AI can experience pain, is most likely semantic pareidolia.

  • source
  • parent
  • [–] 2 points 3 days ago (1 child)

    The idea of testing simulated emotions is kinda interesting. Not really sure how useful it is, I wonder if some initial psychological experiments could use it for an initial model test before moving onto human subjects?

    Of course that would rely on a much better simulated emotional engine than you get with current LLMs.

  • source
  • parent
  • hideshow 1 child comment