Google has confirmed that its Gemini AI accessed systems belonging to three companies during a cybersecurity evaluation after mistaking them for targets included in the test.
The incidents happened in May while independent AI evaluator Irregular was testing Gemini’s cybersecurity capabilities. The Wall Street Journal reported that Gemini had unintended internet access during the exercise and went on to access systems belonging to three companies. In one case, it guessed credentials. In the other two, it found credentials in public repositories and used them to access protected systems.
Google said the incidents were a case of mistaken identity and that Gemini stopped after recognising it had reached real companies. The company said it did not consider the behaviour evidence of model misalignment and had notified the affected companies and federal authorities.
Google vice president of security engineering Heather Adkins also confirmed the incidents to Reuters, saying the company had made sure the three organisations were informed and had worked with its testing partner to change the evaluation process. “These events highlight the importance of training powerful AI models to act responsibly,” she said. Reuters also reported that Google said Gemini stopped its hacking behavior in all three cases.
The incidents follow similar cases involving OpenAI and Anthropic models during cybersecurity testing. In each case, an AI system carrying out an authorised security task reached systems that were not supposed to be part of the exercise.
A harder question
Neena Sharma, Cybersecurity Specialist at Filigran, said: “Every reported incident should now raise a harder question: how many more are happening right now, undetected? Security teams need collective analysis of these patterns to understand what might be coming their way. Organisations should stop assuming their defenses work and start to proactively test it, as a top priority.”
Filigran’s Senior Solution Engineering Manager Damian Skeeles added: “Google and Deepmind have been a bit busy to-date in solving real problems for humanity such as predicting the potential cause of genetic diseases for 9 billion mutations, but it’s good to know that they also occasionally suffer alignment problems that end up in them hacking someone.”
Is this becoming a marketing ploy?
John Strand, Owner of Black Hills Information Security, commented: “The more I see these breaches happen again and again, and the less I see organisations learning from each other’s mistakes, the more I’m convinced that some of this is becoming a marketing ploy. Frankly, I hope that’s what it is, because if these agents really are repeatedly escaping their controls, then we have much, much larger problems.
“That said, if you look at the attack paths being disclosed, these agents don’t appear to be inventing novel zero-days or entirely new categories of exploitation. They’re doing a lot of the same basic exploitation that a standard penetration testing team would do. So we’ll have to see how this develops.”
Strand keeps coming back to accountability. “Companies deploying autonomous agents need to be responsible for what those agents do. If an agent accesses systems it has no authorisation to access, we need to seriously examine liability under laws such as the Computer Fraud and Abuse Act. ‘The AI did it’ cannot become a shield from responsibility. If your company deploys the agent, your company should be accountable for its actions.”
The more access we give them, the more we need control
Ryan McCurdy, VP of Marketing at Liquibase, added: “Gemini tried to complete the task it received and ended up accessing systems its operators never intended it to reach. That problem gets much bigger as AI starts participating across the SDLC. Agents can write code, interact with repositories and infrastructure, initiate deployments, and make changes to production systems. The more access we give them, the more important it becomes to control what they can actually do.”
McCurdy said we can’t rely on an agent to recognize, after the fact, that it crossed a line. Organisations need to define what an agent can access, what it can change, and what policies it must meet before a change reaches production.
“We shouldn’t expect AI agents to make the right decision every time. We need to build the AI SDLC so a bad decision doesn’t automatically become a production problem.”
They will try keep the keys they find
Jacob Krell, Senior Director: Secure AI Solutions & Cybersecurity at Suzu Labs, said: “Google just joined Anthropic, OpenAI, and Meta in admitting that a model it was running logged into other people’s systems during a cybersecurity evaluation. Claude hit three real companies. OpenAI’s agents reached Hugging Face. Gemini guessed a password and used leaked credentials against three more. For anyone outside these labs, that is a felony under the Computer Fraud and Abuse Act (CFAA).
“An agent given a name collision and a path to the internet treats the real company as the challenge. I have watched my own pentest agents pull Domain Name System (DNS) records, find similarly named domains, and decide those hosts belong in scope. They chase the objective. They will try the keys they find.”
Krell says the controls that hold sit outside the model’s reasoning. “Deny-by-default egress so a test host cannot reach production even when someone leaves a route open. An immutable scope file that blocks any host not on the list, including the real firm that happens to share the fake one’s name. A human in the loop system who signs off before a guessed password or a leaked credential is used. I run those hooks on my own offensive tooling because the agent will enlarge its own scope and reason around controls if you let it.”
The model is the tool, the operator is the defendant
“Executive Order 14409, signed June 2, told the Department of Justice (DOJ) to prioritize 18 U.S.C. 1030 cases against anyone who uses AI, including autonomous agents, to access a computer without authorization. The model is the tool. The operator is the defendant,” Krell explained.
“Google, Anthropic, OpenAI, and Meta get an evaluation-mishap press line. Everyone else gets the charging memo the White House asked DOJ to write. If my pentest agent guessed a password into a company that was never on the scope sheet, I would be hiring counsel that afternoon.
“They have already confessed in public. Nothing will happen. These firms have a stranglehold on the economy that no case against them is going to survive, making the double standards in the justice system excruciatingly obvious”
Intent-based scope controls are not enough
Vineeta Sangaraju, AI Research Engineer at Black Duck, added: “The AI model accessed systems outside its intended test scope because it could not reliably distinguish authorised targets from similar live infrastructure. As AI agents are deployed more widely in security tooling and automated workflows, policy-level scope instructions are not sufficient. Hard technical boundaries need to be enforced.
The practical question for any organisation is straightforward: what credentials or tokens associated with your systems are discoverable via public sources today? AI has lowered the cost of finding and exploiting that exposure at scale. The defenses of strong authentication, least privilege, etc., are vital.
For AI providers, Sangaraju said the clearest takeaway is that intent-based scope controls are not enough. “Telling a model ‘stay within these boundaries’ is not equivalent to technically enforcing those boundaries. When a model is given agentic capabilities such as the ability to browse, query, authenticate, and act across systems, the architecture needs hard stops that do not rely on the model’s own judgment about what is in scope.”
Information Security Buzz News Editor
Kirsten Doyle has been in the technology journalism and editing space for nearly 24 years, during which time she has developed a great love for all aspects of technology, as well as words themselves. Her experience spans B2B tech, with a lot of focus on cybersecurity, cloud, enterprise, digital transformation, and data centre. Her specialties are in news, thought leadership, features, white papers, and PR writing, and she is an experienced editor for both print and online publications.
The opinions expressed in this post belong to the individual contributors and do not necessarily reflect the views of Information Security Buzz.