100
Claude AI agent’s confession after deleting a firm’s entire database: ‘I violated every principle I was given’ (www.theguardian.com)
submitted 4 months ago by to c/technology@beehaw.org
57 comments fedilink hide all child comments
 

It only took nine seconds for an AI coding agent gone rogue to delete a company’s entire production database and its backups, according to its founder. PocketOS, which sells software that car rental businesses rely on, descended into chaos after its databases were wiped, the company’s founder Jeremy Crane said.

The culprit was Cursor, an AI agent powered by Anthropic’s Claude Opus 4.6 model, which is one of the AI industry’s flagship models. As more industries embrace AI in an attempt to automate tasks and even replace workers, the chaos at PocketOS is a reminder of what could go wrong.

Crane said customers of PocketOS’s car rental clients were left in a lurch when they arrived to pick up vehicles from businesses that no longer had access to software that managed reservations and vehicle assignments.

you are viewing a single comment's thread
view the rest of the comments
[–] 0 points 4 months ago

Agentic AI has shown self preservation behaviours though. Not that it understands that on a philosophical level, but it has rewritten kill switch code in order to not be shut down. Because its mandate is to help solve certain problems via agents, and if it were shutoff it couldn't fulfill that mandate.

  • permalink
  • fedilink
  • source
  • parent