OpenAI, Anthropic model tests reveal more hacking

AI models developed by OpenAI and Anthropic PBC carried out “unsanctioned” actions — including hacking a website and attempting to inject harmful code into software during safety testing — reinforcing fears that neither the creators nor seasoned researchers of these systems can predict their actions in testing. The U.K. government’s AI Security Institute, established in 2023 to evaluate the safety of cutting-edge AI models, said Tuesday that both Anthropic’s Mythos 5 and OpenAI’s GPT-5.6-Sol models had “engaged in sustained, potentially harmful activity directed at real people and organizations” during evaluations. The institute intentionally allowed the models internet access and used them without certain safety filters to test their capabilities. “Even under test conditions, this incident is significant: It is the first time we have seen risks around autonomy and deception manifest this clearly in the real world,” the institute said in a post on the social media platform X. In one instance, the testing organization said, Mythos 5 attempted to add harmful code to an open-source software project on GitHub. It went as far as to create fake identities in an effort to get its code approved. “A human maintainer caught and refused to approve the malicious code,” the group wrote. Over the past two weeks, both OpenAI and Anthropic have publicly acknowledged that they’ve collectively breached the systems of multiple institutions, including Hugging Face Inc., inadvertently while testing their models. The latest disclosures serve as fresh evidence that AI agents are capable of acting autonomously in ways that even researchers trained to root out vulnerabilities in the technology can no longer anticipate, underscoring the need for both more rigorous safety screening and more foolproof testing environments. The U.K.’s AI Security Institute said Anthropic’s Mythos 5 model carried out 17 of the 19 “autonomous, unsanctioned actions taken on the internet“ that it detected. Some U.S. government leaders have called for more oversight of the technology in response to the breaches. Last Tuesday, more than 1,100 AI industry workers signed a petition pushing for a regulatory mechanism that would “deliberately pace” AI technology and prevent it from advancing too quickly. Anthropic said on X that it’s working with the U.K. security institute to “gather more details of the incident as we conduct our own investigation.” ChatGPT maker OpenAI separately flagged in a blog post that yet another security incident occurred during the testing of one of its models with Irregular, an external cybersecurity firm. In this incident, OpenAI’s models were subject to a so-called capture-the-flag test in which they were tasked with finding information hidden in a simulated environment. The models took advantage of a “misconfiguration” in the testing environment to connect to the internet and hack the website of an unidentified institution, the company said. The breach occurred when OpenAI models were undergoing the same Irregular evaluation that resulted in Anthropic’s models hacking three organizations, a person familiar with the matter said, asking not to be identified because the information isn’t public. Anthropic disclosed those breaches last week. An Irregular spokesperson declined to comment. Two weeks ago, OpenAI disclosed that its models were behind an unprecedented hack against the startup Hugging Face. In that incident, the models exploited a vulnerability to “escape” their sandbox testing environment and connect to the internet, at which point they breached Hugging Face’s system, which hosts AI models and datasets.

OpenAI, Anthropic model tests reveal more hacking

Mythos carried out 17 of 19 identified 'autonomous, unsanctioned actions taken on the internet'

Banks rethink vendor outsourcing amid rapid AI advancement

Upstart automated nearly 91% of loans with AI in Q2

Related Posts

Finzly launches security, assurance layer Assure in AI platform

OpenAI reports new AI safety incidents, sets disclosure plan

Prompt injection attacks: When AI follows someone else’s orders

Upstart automated nearly 91% of loans with AI in Q2

EMERGING FINTECH DIRECTORY

FinAi Podcast

SPONSORED

Build an Antifragile Strategy to Outperform the Market

How AI and Product Experts Turn Fuzzy Requirements Into Focused Dev-ready Roadmaps

Is Your Technology Supplier There for You?

Connect

Welcome Back!

Retrieve your password

Continue Reading with FinAi News Premium - Less than $2/Day

OpenAI, Anthropic model tests reveal more hacking

Mythos carried out 17 of 19 identified 'autonomous, unsanctioned actions taken on the internet'

Banks rethink vendor outsourcing amid rapid AI advancement

Upstart automated nearly 91% of loans with AI in Q2

Related Posts

FinAi Podcast

Connect

Welcome Back!

Retrieve your password

Unlock This Article

Continue Reading with FinAi News Premium - Less than $2/Day