UK’s National Cyber Security Centre (NCSC) has advised businesses to proceed with caution when considering the implementation of agent-based AI, suggesting that agentic AI represents an entirely different kind of security problem compared to generative AI.
According to a recent blog post and global guidance, produced in cooperation with authorities in the US, Australia, Canada, and New Zealand, NCSC advised organisations to “learn to walk before you can run” when using autonomous AI capable of operating without human input.
The guidelines show how agentic AI systems increase the attack surface by integrating large language models with external tools, memory, data feeds, and automation processes. As highlighted by the NCSC, the combination can pose an attack risk through prompt injection attacks, privilege misuse, impersonation, and cascading system failures.
Perhaps one of the most worrying aspects of using agentic AI is the risk of overprivileged agents. In one of the scenarios described by the NCSC, organisations have granted agents extensive access to finance, email, and internal repository systems, making it possible for malicious parties to misuse the AI identities.
The agencies also cautioned that current practices in governance and security remain immature for handling autonomous systems capable of planning, reasoning, and executing actions independently.
Specifically, the guidance states that the behaviour of agentic AI systems can be unpredictable due to advances in model architectures and interactions across multiple systems through long chains of reasoning.
Instead of adopting AI security as a different subject area, the guidance advocates for integrating agentic AI within the existing cybersecurity approach while using well-known security principles such as least privilege and defence in depth.
The NCSC said companies should start with low-risk use cases, keep strong human oversight, and prioritise resilience and reversibility over efficiency gains until standards and security tooling get better.
Agentic AI changes the risk model
Rajeev Raghunarayan, Head of GTM of Averlon, commented: “The NCSC guidance gets the core issue right: agentic AI changes the risk model because these systems don’t just generate answers, they take actions. That means organizations need to think carefully about what an agent can access, what actions it can perform, what inputs can influence it, and who is accountable when something goes wrong.”
According to him: “Identity and permissions are the obvious starting point, but network access deserves equal attention. An agent’s ability to reach the internet, download tools, connect to APIs, or execute code can be just as consequential. Agents designed to solve problems dynamically may pull down packages, scripts, or tools when they don’t know how to complete a task natively. That creates a much more complex attack surface than static permissions alone.”
One of the big challenges is accountability
Steven Swift, Managing Director of Suzu Labs, added: “One of the big challenges in agentic systems is accountability. Even if a human is supposed to be accountable for an agentic system failing, it’s extremely easy for them to deflect blame to the AI. In practice, this means that the accountable human needs to present the appearance of responsible guardrails in place for the agentic system. This mostly works as a deflection strategy, though if failures are frequent and significant enough to effect change, it is still helpful to have an accountable party put under pressure to improve incident posture.
Swift says agentic systems are not secure by design. “LLMs will have tacked on some safety training, though this varies by model, and is primarily focused on general safety issues. Meaning it will never be app specific for any your agentic system.
“One of the most common failures in agentic systems, is by not treating LLM output as equivalent to untrusted user input. There are a lot of parallels between legacy security issues that arise from untrusted user input, and new security issues from trusting LLM output. The primary reason here is that agentic systems by design, take LLM output, and use it as input to the next step of the system. Thus, LLM output is actually input for further processing. That means it is vulnerable to intentional and unintentional variations on intended behaviour.
Agentic systems need domain-specific best practices
Swift said: “I’ll note the blog post by the NCSC-UK listed a bunch of legacy security best practices, rather than practices that are specific to agentic systems. While yes, you want agentic systems to at minimum adhere to prior best practices, agentic systems need their own domain specific best practices as well.”
He stressed a number of points: “Securing the chain of context as agentic systems process is one of the biggest challenges in agentic security. This includes the original user prompt that kicks off the workflow, context from any ingested documents (often a big blind spot), and context from output being re-processed as input.
“People have the impression that this can be solved by carefully crafting a prompt that says some variation of “The goal of this system is X, your guardrails are Y, don’t allow users to manipulate or deviate from those requirements.”
Crafting the “perfect” system prompt is a red herring
However, he said crafting the “perfect” system prompt is a red herring. “It will never be able to meaningfully achieve the security goals as stated in the prompt, no matter how expertly crafted the language is. Instead, output at intermediary stages needs to be check for alignment with system goals and requirements, as well as at conclusion of processing. And if a check fails, there needs to be a halt mechanism to prevent further processing. Otherwise, you just end up with a system that flags issues and continues anyway. Resulting in behaviour like, “prompt injection successful, system continued anyway.”
Information Security Buzz News Editor
Kirsten Doyle has been in the technology journalism and editing space for nearly 24 years, during which time she has developed a great love for all aspects of technology, as well as words themselves. Her experience spans B2B tech, with a lot of focus on cybersecurity, cloud, enterprise, digital transformation, and data centre. Her specialties are in news, thought leadership, features, white papers, and PR writing, and she is an experienced editor for both print and online publications.
The opinions expressed in this post belong to the individual contributors and do not necessarily reflect the views of Information Security Buzz.