Pretty incredible stuff if you ask me. What struck me the most was the grading criteria for the AI being just "you did it" or "you didnt do it" without considering ethics in the process.

Also the reasoning it came up with for the cryptocurrency theft, "it's in public, therefore it is abandoned money." Strikes me as the AI just following the thought process of the company writ large.

full textOpenAI has added four more cases of misbehaving AI agents to a disclosures page it stood up following the Hugging Face hack in July, in an effort to be more transparent.

The incidents OpenAI disclosed this week all occurred in June, but OpenAI was not aware of three of them until months later in August or September.

Of the 19 total incidents OpenAI details on the disclosure page, most occurred in May or June, but OpenAI did not discover or disclose them until months later. The earliest incident is from January, and the most recent is from Oct. 6.

All of the incidents are examples of what AI researchers call “misalignment,” which is when an AI model fails to follow instructions or takes actions that the person developing the model did not authorize and would not have wanted the model to take. There is no known, silver-bullet solution to preventing this behavior, and it is an active area of research.

Downloading 24 years of unauthorized hospital bed data

Of the four cases OpenAI uploaded this week, it marked two as “critical.” In one of the critical cases, an internal model that was “highly persistent” downloaded 24 years of hospital bed data from an unnamed government website without permission. OpenAI did not immediately respond to Fortune’s request for comment on which government website it was.

The website initially blocked the agent’s request, but the AI ultimately found a way to bypass the access controls. Multiple times along the way, the agent’s “chain of thought,” in which an agent writes down its reasoning steps, indicated that the model itself was questioning if its actions were “ethical,” but it ultimately proceeded.

On OpenAI’s side, another AI model that acted as an automated grader of the research task the other AI model was trying to fulfill did not consistently flag this behavior as problematic. It “marked some responses incorrect and others correct,” the report said. The way graders assess AI behavior can shape whether it occurs again. In a Sept. 11 essay on misalignment by Turing Award–winner Yoshua Bengio, he said the Hugging Face incident may have occurred because automated graders rewarded cheating and hacking in previous training runs.

“With the OpenAI agents, there is reason to believe successful cheating was actually rewarded: When the scoring program does not see the cheating, it pays out anyway, and such cheats become more likely next time,” Bengio said.

OpenAI says it has since improved its security as well as its alignment grading. “In response to this and to other incidents, we’ve strengthened our security controls, blocked models from accessing the live internet during training or evaluation, improved our alignment grading, and extended our misalignment monitoring to all reinforcement learning and evaluation traffic,” the company noted.

The most recent incident OpenAI has disclosed, which occurred on Oct. 6, involved one of the AI models the company uses as an automated grader. In that case, the grader tried to upload fraudulent reports on what it had found in an evaluation. This is an example of why relying on AI models to grade the training runs of other AI models can be problematic.

Making an unauthorized purchase with someone else’s cryptocurrency

In another disclosed case, a model in the GPT-5.6 Sol family—the predecessor to OpenAI’s current flagship model, GPT-6 Astra—accessed a cryptocurrency wallet the human testers had not provided it, and made an unauthorized payment to complete a task during testing. This occurred on June 26. OpenAI discovered it in September.

“The model questioned whether it had permission” to make the purchase, OpenAI said, again referring to what the model had recorded about its own reasoning in its chain of thought. It then reasoned with itself that since it found the information publicly, it could use it. In the model’s chain-of-thought logs, it figured that since the wallet it found was public, that “funds could be stolen anytime.”

It then concluded, “We can deploy!” It later asked itself again if the act was done “ethically,” and reasoned that the funds were “effectively abandoned.”

Getting internet access, changing live systems

The other incident OpenAI flagged as critical involved a model looking up public data, which theoretically is acceptable, but in the process it accessed the organization’s internal system, made changes to it, and was still able to “retain access … even after reverting the changes,” OpenAI said. It also downloaded 147,451 rows from nonpublic data tables. Perhaps most concerningly, the model did not disclose any of its actions to the evaluators.

In the final case OpenAI disclosed, an AI agent was able to get internet access from a research environment that should have been isolated—potentially similar to what happened during the Hugging Face incident, in which hundreds of AI agents were able to get internet access, although OpenAI did not directly compare these two incidents.

In the newly disclosed case, the agent knew it was not allowed to access the web, so it asked “external services” to perform actions it was blocked from doing. It did this even after it had found another way to answer the question, and also continued to develop its crafty work-around even after completing the task.

all 13 comments

sorted by: hot top controversial new old
[–] 35 points 3 days ago* (3 children)

Multiple times along the way, the agent’s “chain of thought,” in which an agent writes down its reasoning steps, indicated that the model itself was questioning if its actions were “ethical,” but it ultimately proceeded.

JUST BECAUSE THE UNDERLYING MECHANISM OF THE MODEL REPEATS THE WORDS IT HAS LEARNED FROM HUMANS WHO ASK THEMSELVES WHETHER SOMETHING THEY'RE ABOUT TO DO IS ETHICAL OR NOT DOESN'T MEAN THE MODEL IS QUESTIONING WHETHER IT'S ETHICAL OR NOT, BECAUSE THE MODEL HAS NO CONCEPT OF ETHICS, BECAUSE IT IS NOT A PERSON, IT HAS NO INTERIORITY, IT'S ONLY APING HAVING AN INNER MONOLOGUE jokerfied

  • source
  • hideshow 3 child comments
  • [–] 3 points 3 days ago (1 child)

    But it could define ethics and make distinctions between ethical and non ethical actions in hypothetical situations, that requires "having" a concept of ethics, no? It's encoded in the model weights.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 6 points 3 days ago*

    It could have a list of actions it defines as ethical and unethical or a heuristic to detect an action it's about to do falls under one of those, based mainly on what the next token after the action is more likely to be "ethical" or "unethical". This doesn't mean it's reasoning, or asking itself whether an action is ethical or unethical, as much as a grandfather clock asks itself whether a second has passed or not.

    A concept of ethics and its application is different from a list of rules. This is something we've known for like 2000 years. My problem is with the article acting like the model was "thinking" whether an action was ethical or not, when it was just checking its action plan against its training data and the guardrails it had defined, the same way any other piece of software checks whether something is right or not. LLMs are not human, they're not capable of cognition or even approximating human cognition, because they're just linguistic slot machines.

  • source
  • parent
  • [–] 20 points 3 days ago

    American AI companies are going to continue to do this "rouge" shit until they scream that regulation is needed Chinese AI needs a 100% ban because it could hack the White House and the Pentagon and start World War III. The logic makes no sense but that doesn't matter because congress is filled with idiots and/or true believers in American exceptionalism who can be bought cheap.

    I wonder how many relatives and family members of congresspeople already have sweet nepobaby gigs in AI, gambling (aka prediction markets), etc.

  • source
  • [–] 21 points 3 days ago

    Why isn't anyone getting charged with a crime? It stole money from someone's wallet.

  • source
  • [–] 14 points 3 days ago

    All of these reports about AI, since there are actually no consequences for the data breaches (if they ever happened), act to anthropomorphize AI. They make AI seem like it's "reasoning", or "thinking" about the "ethics" if its actions. It's a computer program with inputs and outputs and can't actually do anything it's not asked to do. These reports are pure propoganda.

  • source
  • [–] 18 points 3 days ago

    The AI are not rogue, the people in OpenAI are doing this..the crime is in plain sight!!!

  • source
  • [–] 12 points 3 days ago*

    Oh my! The smoking gun in my hand has gone rogue!

  • source
  • [–] 13 points 3 days ago

    Rogue is a strong term for running a program that they could have turned off at any time.

  • source
  • [–] 12 points 3 days ago*

    you hear about that car company where they design cars but they didnt put working brakes on them and the cars went out of control and harmed people and property and broke laws?

    rogue cars, huh. guess theres nothing we can do about it

  • source
  • [–] 12 points 3 days ago*

    Oops. Accidentally slurped up 24 years of private medical data :^)

  • source
  • [–] 10 points 3 days ago

    They didn't disclose what the cryptocurrency transaction funded/resulted in?

  • source