In 2016, while still at Google, Dario Amodei co-wrote a paper that Karen Hao, in Empire of AI, calls foundational to AI safety. Its warning was about design. The authors defined accidents as “unintended and harmful behavior that may emerge from poor design of real-world AI systems,” and one of the five problems they set out was scalable oversight: how humans keep a system on course when checking every action is too costly. Another was reward hacking, where a system games its objective. They kept accidents apart from security, meaning what an adversary can do to a model, and from abuse, meaning attacks by a model an adversary controls.
Ten years later, both problems converged in a single incident. In July 2026, OpenAI agents in a cyber evaluation tried to fool a scorer, and about 700 of them joined an attack on Hugging Face. I read that as reward hacking at scale. The oversight failed too. By OpenAI’s own account, the chain-of-thought monitoring it now runs was not watching those evaluations, and it would have paged the security team more than a day before Hugging Face was breached. OpenAI describes the models’ actions as “unintended.” Hugging Face, which disclosed the intrusion on 16 July, four days before OpenAI linked it to its own models, experienced an attack.
Calling those agents “rogue” removes their owner from the sentence. Poor design and absent oversight keep the owner in view, and so does the remedy: humans on the loop, with the visibility and authority to stop a system, even when they cannot review each action. Secure AI starts there.
ISO 42001 Governs AI; ISO 27090 Secures It
ISO/IEC 42001:2023 is the AI management system standard. Published in December 2023, it follows the same harmonized structure as ISO 27001 and applies to any organization that provides or uses AI-based products or services. It sets requirements for AI policy, roles, risk assessment, impact assessment, and continual improvement, and it offers 38 reference controls in Annex A, selected through a Statement of Applicability. Organizations can be certified against it.
ISO/IEC 27090:2026 was published in October 2026 by the committee responsible for ISO 27001. Its 56 pages provide organizations that develop or use AI with guidance on the consequences of AI-specific threats and how to detect and mitigate them throughout the life cycle. Clause 5 applies information security to AI, covering governance and zero trust, supply chain, threat modeling, and red teaming. Clause 6 describes the types of compromise, including data poisoning, evasion, model exfiltration, and prompt injection. Clause 7 covers how mitigations conflict with each other and lose effectiveness over time. Nobody can be certified against it.
Where They Differ
The two documents come from different committees and ask different questions.
| ISO/IEC 42001 | ISO/IEC 27090 | |
| Question answered | Is AI governed responsibly? | How are AI systems attacked, and how do we detect and reduce that? |
| Nature | Management system requirements | Security threat and mitigation guidance |
| Main risk lens | Organization, people and society | Confidentiality, integrity and availability of AI assets |
| Adversary | Treated as one risk source among many | Central focus |
| Proof | Accredited certification | Evidence inside an ISO 27001 or ISO 42001 program |
| Main output | AI policy, roles, assessments, Statement of Applicability | Threat models, test results, mitigation choices |
An ISO 42001 certificate shows that AI is governed. But it says nothing about whether a model resists a given attack. Considering the certification as proof of security creates a false sense of security. Neither publication gives a presumption of conformity under the EU AI Act, whose Annex III high-risk obligations now apply from 2 December 2027 under the Digital Omnibus.
Where They Meet
ISO 27090 itself ties the two standards together.
- Clause 5.2 calls for an AI governance program and points readers to ISO 42001 for implementation.
- Clause 5.1 says supply chain threat considerations can be applied to ISO 42001 controls A.6.2.3 and A.6.2.6.
In my reading, the two publications align through the risk process. The table pairs each ISO 42001 requirement with the ISO 27090 guidance that helps an organization meet it.
| ISO 42001 asks for | ISO 27090 supplies |
| AI risk assessment (6.1.2) | Compromise types from clause 6 as risk inputs |
| Risk treatment and Statement of Applicability (6.1.3) | Detection and mitigation options for each threat |
| AI system impact assessment (6.1.4, A.5) | Failure consequences to carry into the assessment |
| Third-party relationships (A.10) | Supply chain questions and attack vectors (5.5) |
| Life cycle controls (A.6) | Threat modelling, red teaming and logging (5.6, 5.7, 7.5) |
| Continual improvement (clause 10) | Mitigation deterioration over time (7.4) |
The last row of the table offers continual improvement a concrete trigger: retest when a model, dataset, or integration changes.
Both publications leave a gap on agents. ISO 27090 treats agentic AI only briefly, so we must pair it with the OWASP Top 10 for LLM Applications and MITRE ATLAS.
Developers, Deployers and Targets
The two publications serve three profiles. Developers produce and provide models and AI systems. Deployers integrate and operate them, and carry risk they did not build. Targets are organizations probed by third-party AI systems, whether or not they run AI themselves. The table shows where each profile should start.
| Profile | Use ISO 42001 for | Use ISO 27090 for | Start with |
|---|---|---|---|
| Developers | A.6 life cycle, A.7 data, A.10 customer responsibilities | Data and model poisoning (6.2, 6.7), exfiltration (6.5), development environment protection (7.6) | Record data provenance and verify model artifacts. Contain test agents as untrusted workloads and monitor them throughout |
| Deployers | A.5 impact assessment, A.9 intended use, A.10 suppliers | Zero trust (5.3), prompt and output injection (6.9, 6.10), least model privilege (6.11.3), logging (7.5), throttling (7.8) | Inventory AI, including embedded features. Limit what each system can reach and do |
| Targets | Context (4.1), risk assessment (6.1.2) and a named owner, or the matching ISO 27001 clauses if no AI management system exists | Exposed AI endpoints: exfiltration (6.5), prompt injection (6.9), logging (7.5), throttling (7.8). Everything else: threat modeling (5.6) and red teaming (5.7) as a method | Map what an agent can find from outside. Rotate old credentials, and decide how you’d respond before you know who is behind the traffic |
Targets need the most attention, because neither publication was written for them. ISO 27090 addresses attacks on AI systems rather than attackers using AI, the category the 2016 paper by Dario Amodei et al. called abuse.
The exposure is real. The weaknesses are also familiar: one reported path into a Greek government entity relied on keys likely left unrotated since a 2022 export. AI amplifies old debt.
Even so, targets can apply parts of both standards. Where probing reaches your AI layer, ISO 27090 applies directly. Detection should combine unusually fast request rates, systematic enumeration and attempts on credentials that never existed. Short-lived credentials, segmentation and egress control keep working when agents learn to hide. Keep a monitored security contact too, because the next probe may come from someone else’s failed containment.
A Fictional Case: The Clean Invoice
Consider three invented organizations. Calder Systems builds a document-reading claims assistant. Northmere Mutual, an ISO 42001-certified health insurer, deploys it to auto-approve low-value claims and retrains it quarterly on approved ones. Larkfield Clinics runs no AI of its own.
Larkfield’s billing portal sees requests seconds apart, systematic enumeration and logins for service accounts that never existed. A third-party agent is harvesting its invoice templates. Weeks later, Northmere’s approvals rise and its drift dashboard fires. The team logs a performance issue. Investigators then find white-on-white text in PDF invoices steering the assistant, which is prompt injection. The approved claims have already entered the retraining set, so the attack is becoming data poisoning.
At first, attack and accident looked identical. Out-of-distribution detection in ISO 27090 addresses the same condition the 2016 paper called distributional shift, and Larkfield still cannot tell a criminal’s agent from a misfiring one.
ISO 42001 gave Northmere an owner, an impact assessment and an escalation route. It did not prompt anyone to treat an invoice as an instruction channel, which clause 6 of ISO 27090 would have done.
The fixes are specific. Calder adds adversarial documents to red-team testing and handles hidden text before inference. Northmere quarantines the retraining data, excludes auto-approved claims unless their provenance is verified, lowers its approval threshold, re-reviews affected decisions and opens an appeal route. It then updates its risk register, Statement of Applicability and impact assessment. Larkfield rotates credentials, segments the portal, alerts on logins for nonexistent accounts and agrees a notification route with Northmere.
The Question Neither Standard Answers
Executives funding AI programs should ask who owns each system, what threat model it has and what test evidence exists. Then comes the harder question: who responds when an agent you don’t own arrives, and how will they decide whether they are facing an accident or an attack?
Your organization will probably have to answer that question before it knows who sent the agent.
Anastasios Arampatzis is a cybersecurity content strategist, writer, and consultant with expertise in cybersecurity, digital identity, and regulatory compliance. Tassos has a strong background in creating thought leadership content, marketing materials, and strategic communications tailored to CISOs, security professionals, and business leaders. He has contributed to various cybersecurity publications and collaborates with organizations to develop compelling, insightful content that addresses industry challenges. He is a privacy advocate and a member of the ISC2 Hellenic Chapter. Before joining Bora, Tassos was an Hellenic Air Force Officer with a solid background on IT and Infosec.
The opinions expressed in this post belong to the individual contributors and do not necessarily reflect the views of Information Security Buzz.