you are viewing a single comment's thread
view the rest of the comments
[–] 0 points 2 hours ago

Agents definitely do act in unexpected, unprompted ways, if they didn't then alignment wouldn't be a worry. Did you read about the hugging face hack? There was nothing in the prompt about hacking hugging face according to the third party audit but it did it any way.

Even smaller cases like Claude deleting a bunch of data when the initial prompt was "fix this bug" or whatever is an agent acting outside it's prompt. It's still the users fault because they shouldn't be giving Claude that kind of access in the first place, but it's not like they commanded Claude to delete the data.

  • source
  • parent