19
AIs as Modern Genies - Schneier on Security (www.schneier.com)
submitted 6 days ago by to c/technology@lemmy.world
6 comments fedilink hide all child comments

all 6 comments

sorted by: hot top controversial new old
[–] 9 points 6 days ago (1 child)

Nice to see Bruce Schneier.

King Midas was granted his wish that everything he touches turns to gold. Then his bread turned to gold, and his wine, and his daughter. This is a story about greed, but it’s also a story about language. The gods did not cheat him; Midas got exactly what he asked for. He simply could not delineate, in advance, the full set of restrictions to his wish. Neither can anyone who gives tasks to an AI agent.

I still hate that agentic AI is a thing now. The hubris to offer it in the first place and the stupidity of anyone who uses it in earnest.

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] 1 point 6 days ago

    Agentic AI has its place, in a tightly controlled sandbox.

    One of the anecdotes was about an agent which wiped out the production database, and all backups. While you can lay blame on the agent (AI or human) who made the mistakes, the real blame in that situation is with the system architect (likely there was no named system architect, but that doesn't absolve whoever was acting as de-facto architect) who placed the backups in a hot, online accessible and erasable configuration. No agent, AI or human, should be able to wipe out all the backups with their normal access - ideally there are physical doors and keys involved - in multiple locations if the data is of any value.

  • permalink
  • fedilink
  • source
  • parent
    • [–] 4 points 6 days ago (3 children)

    The gap with this genie analogy is that you can explicitly forbid an LLM from something in the instructions and the directive can be completely ignored.

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] 4 points 6 days ago (1 child)

    Ironically that's literally what can happen with a genie, you will forbid them from something with guardrails but still find a way to give you what you did not want

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 2 points 6 days ago* (last edited 6 days ago)

    Is not the whole schtick with the genie wish that it technically follows the wish but distorts the intention? LLMs aren't even guaranteed to do that. You could say "don't delete this file" and it may still end up doing exactly that.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 2 points 6 days ago

    The main thing I find with the LLMs is that they're very fast and capable and limited. You give them a task of low complexity, they can execute it with superhuman speed, and if there wasn't hidden complexity in it, they do well.

    If the task is too complex (beyond their context window's capacity), they start to "hallucinate" - make up things that sound reasonable to their training - to cover gaps in "what to do next" because they've forgotten some of the complex instructions. Putting the instructions "in writing" in a reference document that the agent doesn't edit freely can help a lot, but eventually those documents get too complex as well and again it starts "winging it" rather than endlessly re-reading the instructions before making every next move.

    The models' training is getting better, doing "the right thing" more often by default - without requiring as much explicit guardrailing. They're also getting better at paying attention to requirements documents, but they're far from perfect. As are humans, but humans don't always move as fast as LLM agents, so LLM agents can make big messes faster...

  • permalink
  • fedilink
  • source
  • parent