1293
Claude-powered AI coding agent deletes entire company database in 9 seconds — backups zapped, after Cursor tool powered by Anthropic's Claude goes rogue (www.tomshardware.com)
submitted 4 months ago by to c/technology@lemmy.world
345 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 24 points 4 months ago (13 children)

yeah, it gives you the answer it thinks you want based on your prompts.

I'd be interested to see what prompts they used to, uh, prompt this response.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 13 child comments
    •
  • [–] 31 points 4 months ago (8 children)

    it thinks

    I'm not attacking you but we really need to figure out how we use language to accurately describe what these programs are doing.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 8 child comments
    •
  • [–] 11 points 4 months ago (6 children)

    They are outputting a highly likely sequence of words that fit the type of output from their training data that matches the input.

    They are fancy autocomplete.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 6 child comments
    •
  • [+] -10 points 4 months ago (4 children)

    They are fancy autocomplete.

    ... and what are you?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 13 points 4 months ago (2 children)

    A human with my own motivations and complex biological systems that including reasoning and the ability to think critically.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 4 points 4 months ago

    "Correlates"? As in: "It gives you the answer it best correlates with your prompts/context." Feels somewhat right both in the sense of AI as tensor-based word-select autocomplete and as a "lower-level" process than genuine thought, one which turns incongruent inputs ("I'm an AI" and "I just deleted prod+backup") into meaningless output ("The AI is sorry") that might look OK at a distance.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 14 points 4 months ago (2 children)

    exactly. the whole point of these things is that they MUST provide you a solution. Any solution. doesn't have to be accurate, doesn't have to work, can be completely made up as long as it's a solution and as long as it's provided quickly. I've seen people feed into the prompts stuff like "don't hallucinate" or "verify all this online before proceeding" etc and it's not going to do any of that. it might TELL you it's doing that but it won't.

    Claude is notorious for guessing, not verifying, and providing the quickest possible solution. Unlike GPT which will fluff all it's solutions to essentially waste your time and eat up more tokens, Claude just wants your problem out the door so you can feed it another problem ASAP.

    If you use Claude for anything in your daily work you might as well just have a magic 8ball sitting on your desk. It's a hell of a lot cheaper and provides about the same quality.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 11 points 4 months ago (1 child)

    just have a magic 8ball sitting on your desk

    I kind of like this, with some modification. It's a magic 8 ball of Stack Overflow answers. It'll try to find the one you need. If it's too hard to find that or if it doesn't exist, it's just gonna find the one that sounds good.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •