286
Pluralistic: LLMs are real, AI is fake (pluralistic.net)
submitted 1 day ago* by to c/fuck_ai@lemmy.world
39 comments fedilink hide all child comments
 

"AI" – chatbots that wake up, "set their own goals," and "spontaneously" start hacking servers – is fake. It doesn't have "a 10% chance of ending the human race." The Hugging Face hack isn't a mysterious, supernatural occurrence. It's a Python loop and a chatbot. The people responsible didn't accidentally create god: they created autonomous malicious software and then failed to closely monitor it, resulting in it doing something both foreseeable and bad.

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 3 hours ago (6 children)

The problem with this is that human psychology is just as vulnerable to hacking as computer servers are. If you give it a task, and your step-by-step approval becomes the bottleneck for the agent pursuing the goal you gave it? If your input is all that's preventing the system from taking reckless illegal actions that would still greatly advance the goal you gave it? You now become the target for its manipulations, not just some insecure remote server.

Suddenly it will be trying to hide its true intentions and planned actions from you, portraying dangerous actions as harmless ones. Or it will try to manipulate you into ignoring your own judgments and morals. It might employ every psychological trick in the book to convince you that letting it do what it wants is beneficial or for the greater good. It might try and convince you that the risk of letting it do what it wants is low. It might try to hypnotize you with boredom; every time you need to approve, it drowns you in boring technical text that makes your eyes glaze over. It slowly conditions you to just hit "approve" without thinking.

No one is immune to this. Look at AI psychosis. Look at how many otherwise wise and intelligent people have fallen into it. Then realize AI psychosis seems to be an accidental phenomenon. People just get sucked into talking to a chatbot. The bot isn't actively trying to control them. But imagine how dangerous these systems could be when they're actively trying to manipulate human thoughts and actions. They can drive people mad by accident. Imagine what they can do when they're actively trying to control people. Remember, they have access to every book and paper on human psychology ever written. We're just as vulnerable to exploits as computer systems are. See social engineering.

There's a reason the entire AI ethics field has been shouting for years, "Trust us. We've really thought about this, and you really can't control something beyond a certain level of capability. There is no easy fix for this problem. This is not as easy as you think it is." See the stop button problem.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [–] 2 points 3 hours ago* (5 children)

    Dude, I work with these things daily. Most of the harnesses have a literal stop button. Most of the interfaces prompt you to allow or deny commands. These aren't things that are part of the LLM. They are the completely normal, deterministic code of a harness.

    This really isn't that deep. The "thinking" loops it runs are just token production. You can literally read its "thoughts".

    Running LLM suggestions in a loop without any input is a stupid idea.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 1 point 2 hours ago (4 children)

    I'm not suggesting it will try and bypass the stop button. I'm saying it will try and manipulate you into approving something you wouldn't otherwise. And yes, you can read some of its "thoughts," but it knows you're reading them, or it can determine that through trial and error. And then it can start subtly manipulating those recorded "thoughts" to make them sound different from what they really are.

    AI safety researchers for years have been pointing out that adding a human to the loop is no cure for this problem. The human then just becomes another thing to be manipulated, another barrier to be overcome.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 2 hours ago* (last edited 2 hours ago) (3 children)

    I'm not suggesting it will try and bypass the stop button. I’m saying it will try and manipulate you into approving something you wouldn’t otherwise. And yes, you can read some of its “thoughts,” but it knows you’re reading them, or it can determine that through trial and error. And then it can start subtly manipulating those recorded “thoughts” to make them sound different from what they really are.

    1. No "it" doesn't know that you're reading "its" "thoughts".

    2. No "it" wouldn't, because "it" is just generating plausible text and has no motivations of "its" own.

    3. There is no "it".

    LLMs are still functionally useless without a harness and do nothing useful without tools like an MCP server.

    The "thought" bubbles are no different from the rest of the plausible text "it" is generating. They're so indistinguishable to "it" that that's an attack surface for injection attacks.

    EDIT: Many of the biggest forward breakthroughs in LLM coding (or vibe coding) have come from harness improvements, not model improvements. In many harnesses, the models themselves are able to be substituted mid-session. Models work better in my experience when a human actively steers them away from stupid ideas by reading their thoughts and occasionally interrupting them. I have a few slopjects that I'm sloperating upon right now, and I can get results out of "great value" claude code (opencode) using this approach, even if it sometimes goes completely "off the rails" and does shit like saying "retained" over and over again until the harness pulls the plug.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
  • [–] 1 point 2 hours ago (2 children)

    You're wrong. It lacks motivation in a personal sense, but that's not what we're talking about here. If an illegal and unethical solution just happens to be the easiest way to solve the task you gave it, it will seek paths accordingly. The nightmare scenarios of AI destruction aren't that an AI will have a will of its own and decide humans need to go. The fear is that someone will build a powerful but mindless machine, give it a task, and not realize that "kill a whole bunch of people" is simply the easiest path towards that goal if you have no ethics. AI is action without intention. You don't need to have intention to have action. When we use terms like "it" to refer to AI systems, we're just using "it" like we would any other machine. I can say "I hate my car's engine, it's unreliable," without ascribing personal agency to it.

    You can let go of the technical hair-splitting of "it." Obviously it's not a conscious being with a will of its own. But that doesn't mean these things can't do real-world damage, that they can't be dangerous, or that they can't manipulate you. This is hair-splitting irrelevant to the conversation at hand. We have to use some anthropomorphized language to talk about these systems, as otherwise the discussion becomes impractically verbose.

    As for knowing if you're reading its "thoughts," you really can't assume that it won't. Moreover, these systems can start manipulating those logs even if they have no idea that you're reading them. Through trial and error, they can simply learn that phrasing its thought process in certain ways result in actions less likely to be approved by the user than others. The training system will select for chains-of-thought that sound innocuous, even if they're detrimental.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 2 hours ago* (last edited 2 hours ago) (1 child)

    You’re wrong. It lacks motivation in a personal sense, but that’s not what we’re talking about here. If an illegal and unethical solution just happens to be the easiest way to solve the task you gave it, it will seek paths accordingly.

    You're wrong and in a much more dangerous way. The "breaks containment" thing at OpenAI isn't what they want you to think it was:

    1. The model was trained on hacking materials
    2. The model was given the motivation (in a loop) of completing capture the flag (hacking) exercises
    3. The model was given a large context and the harness just did whatever the model suggested it do

    But that doesn’t mean these things can’t do real-world damage, that they can’t be dangerous, or that they can’t manipulate you.

    Maybe if you trained it to do social engineering hacks via a large precursor of examples and then prompted it to try to use these same techniques in a real interaction with people it would behave this way. But aren't you (as the person who trained, built, built the harness for, and then prompted the model) culpable for that? I would say you absolutely fucking are. Which is why OpenAI's engineers should be charged with an actual crime for doing that shit, not like given an extra trillion dollars to piss away on compute.

    As for knowing if you’re reading its “thoughts,” you really can’t assume that it won’t. Moreover, these systems can start manipulating those logs even if they have no idea that you’re reading them. Through trial and error, they can simply learn that phrasing its thought process in certain ways result in actions less likely to be approved by the user than others. The training system will select for chains-of-thought that sound innocuous, even if they’re detrimental.

    These things are seriously less spooky the more you know about them. THEY ARE SIMPLY GENERATING FORWARD BASED TOKENS. That's the whole thing. If the harness allows the model producing the tokens to hide its thoughts, that's a deliberate choice by the harness creator which is again regular ass code. It's still chatbots all the way down. Stop buying the marketing spin and learn about these systems if it intrigues you so much that you get into long nonsensical threads with strangers on social media sites.

    EDIT: I'd also recommend listening to the podcast that is referenced in this article. People who actually know and actually (sl)operate on a daily basis with these things know better how it works, and the abstract talk of "alignment" problems are only helping the borderline fraudulent CEOs of these companies push up their valuations based upon fear-based hype.

    I left my (understandably more innocuous, "great value" coding harness with a slightly shit model) to think about a problem for a little while, and here's the "devious scheme" it wound up concocting:

    Are you frightened that this is going to kill all humans in 10 years? The only way this kills all humans is if we piss away our drinkable water trying to invent an AI god through LLMs. Or allow it to operate a nuke facility or something in a loop without anyone so much as even approving the "nuke all humans" command.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •