AI is changing the way software is being developed. From generating code, helping developers make sense of new frameworks, reviewing pull requests, and even suggesting solutions for existing vulnerabilities, AI is playing an increasingly active role in the software development process.
There is an upside here, too. AI is speeding up development, eliminating redundant efforts, and helping organisations fix problems faster. It also raises serious questions about the quality of code, its security, accountability, and trust. When an AI writes or patches the code, who will ensure that it is indeed secure? And, are the current application security practices adequate in this brave new world of programming?
To explore these questions, we asked a panel of security experts to share their views on how AI is reshaping secure software development, where organisations should embrace automation, where human judgment remains essential, and what security teams should prepare for as AI-generated code becomes increasingly common.
Is AI a leap forward, or a risk?
AI is now capable of writing, reviewing, and even patching code. When asked if this represents a genuine leap forward for secure software development, or if risks are being introduced that organisations have yet to fully understand, Gary Hibberd, Fellow of the CIISec, says it’s both.
“It represents a genuine leap forward, but not necessarily a guaranteed leap forward in security. AI can help developers produce code more quickly, create test cases, identify common weaknesses and suggest fixes within seconds. Used properly, it can reduce the time between discovering a vulnerability and remediating it. There is already evidence that AI-assisted security tooling can accelerate vulnerability detection and remediation. GitHub reported that, during 2025, the average time taken to fix vulnerabilities across its ecosystem fell from 37 to 26 days.”
Speed and security are not the same thing
However, speed and security are not the same thing, he adds. “AI-generated code may be functional and still be insecure. Research evaluating thousands of AI-generated code samples found vulnerabilities including hard-coded passwords and path-traversal weaknesses, even where the code successfully passed functional tests. In other words, code can appear to work perfectly while still creating serious security exposure.
“There is also a risk of ‘automation bias’ where developers may trust a confident-looking AI response without fully understanding the code, its dependencies or the assumptions behind it. For me, this is the biggest risk we face. AI can reproduce insecure patterns from its training data, misunderstand the context in which the software will operate or fix the immediate vulnerability while introducing a different weakness elsewhere.”
AI writing code is not the risk
Anastasios Arampatzis, Account Manager at Bora, says AI writing code is not the risk. “AI writing code inside an undisciplined pipeline is the risk. Every flaw already living in an organization’s SDLC, weak requirements, thin testing, and undocumented dependencies, gets absorbed and reproduced by the model at scale. Fast, confident, and wrong. Mature software engineering was always the prerequisite for secure software. AI just removes the excuse for skipping it.”
Once again we’re at the mercy of ‘forces of economics’
Hibberd says our drive for increased speed and reduced costs are also a huge risk, and one that cannot be owned by the technical team. “Once again we’re at the mercy of ‘forces of economics’ as the board drives for faster delivery in the face of economical and market pressures. The message to the board is simply this; development without safeguards is a one-way-street to disaster.
“The opportunity is of course significant, but organisations should treat AI-generated code in the same way they would treat code supplied by an unknown third party: useful, potentially powerful, but never inherently trusted.”
The gap between velocity and explainability is the risk most orgs haven’t priced in yet
Chloe Messdaghi, Founder & Principal Advisor at Thornbridge Advisory, says AI writing, reviewing, and patching code isn’t hype anymore. It’s already in our pipelines. “The question is whether we’re building on solid ground or just moving fast toward a new class of problems we haven’t named yet.”
She believes it’s both a leap and a risk. “AI-assisted development and remediation are real accelerants. Faster triage, faster fixes, fewer humans burning hours on repetitive review work. But speed isn’t the same as assurance. We’re now shipping code where the ‘author’ can’t be interrogated the way a human engineer can. Nobody can ask the model why it made a specific patch decision the way you’d ask a senior dev in a code review. That gap between velocity and explainability is the risk most orgs haven’t priced in yet.”
One of the greatest risks is treating the AI dev as an experienced developer
Ross Moore, an Information Security Researcher, says if the risks are taken into account, it’s a great leap forward. “The AI code-generating capabilities are the equivalent of a junior developer, if the AI dev is properly vetted for reasonable and appropriate accuracy. One of the greatest risks is treating the AI dev as an experienced developer. AI’s ability to replicate, emulate, mimic, and detect trends and patterns is unmatched (think the Imitation Game); but it can’t reason, adjudicate, or understand. Human developers demonstrate these latter qualities, adapting quickly to business requirements and making original and relevant judgments.”
AI doesn’t grow up (or get a free pass)
AI doesn’t grow up, Moore adds, it only learns from what it’s given to train on. It can think faster, but not think better.
“As an extra dev on the team, the AI dev needs to be subject to the same controls as all other devs: secure code is expected, traceability, authentication enforced, no cowboy coding, validation, etc. Don’t trust it more just because it’s a machine, and it doesn’t get a free pass.
“An important determination is where the AI resources get the information. Devs need, to some degree, to determine if the code they’re developing is original, or if AI is stealing from others. This lack of insight could result in intellectual property theft. If a team has a streamlined process for reviewing code according to their business needs, then the proper AI tools will prove beneficial in the pre-production phases.”
Treat AI as an accelerator and not an authority
Javvad Malik, Lead CISO advisor, KnowBe4, believes that AI can be a genuine step forward for secure software development, but it should be treated as an accelerator and not an authority. “The danger is organisations trusting plausible-looking output without understanding its assumptions, flaws, or impact. This is why human oversight remains essential for risk acceptance, production changes and security exceptions.”
The line between automation and human oversight
As AI becomes more involved in developing and remediating software, where should organisations draw the line between automation and human oversight?
Moore says human oversight needs to remain for all of it. Automation has been around for a long time, and even before AI, proper human oversight was required for vetting, implementing, managing, and maintaining any resources: the rules, restrictions, timings, and suchlike. All needed to be set up by people. “It’s the same with AI. Low-risk tasks (such as dependency updates, documentation, simple refactoring) are good candidates for automation. High-risk tasks (authn/authz, cryptography, IaC affecting production) require human-in-the-loop.
“The higher the potential impact of failure, the greater the need for human accountability. In some recent events, companies have been held responsible for their AI agent and MCP actions. Just letting the robot do its thing didn’t get the company off the hook for the repercussions.”
People on, not in, the loop
Arampatzis says expecting a human to review every AI-generated commit or patch is not realistic, and pretending otherwise sets teams up for failure. “The workable model puts people on the loop rather than in it. Defined checkpoints, verification gates, and clear ownership of what gets merged, deployed, and rolled back. Automation handles volume, humans hold accountability. Developers are now the architects.”
Own the repetitive, well-understood stuff
For Messdaghi, automation should own the repetitive, well-understood stuff: linting, known-CVE patching, low-risk refactors. “Anything touching authentication, access control, cryptography, or core business logic needs a human in the loop before it merges, and that human needs to be genuinely accountable, not just clicking approve. The line isn’t ‘how good is the AI’. It’s ‘how expensive is being wrong here’. High blast radius means a human signs off, full stop.”
Adapting DevSecOps and application security strategies
When it comes to how security teams should adapt their DevSecOps and application-security strategies to keep pace with AI-assisted software development and automated vulnerability remediation, Hibberd says security teams must recognise that the volume and speed of software change may increase dramatically.
“Traditional application-security processes, particularly those reliant upon occasional manual reviews, will struggle to keep up.”
Strong leadership from security and development teams
Hibberd says this is going to require strong leadership from the Security AND the Development team because they will need to push back against the drive for speed over security. “But we should start with the basics, like having good AI policies and procedures that include risk assessments and impact assessments. These things never go out of fashion, no matter how the AI develops.”
Moore adds that the new AI dev has arrived, so make sure to expand secure coding standards to it, including AI-assisted development, define acceptable AI use cases, and require traceability of AI-generated code.
The goal is better security, not greater AI adoption
“Because velocity has increased, security validation needs to speed up. Strengthen automated validation with activities and tools such as more DAST, SAST, and SCA. Human governance is a must. Vet all proposed vendors according to the risk presented (AI has required vendor reviews to be much more rigorous to ensure alignment with international regulations); monitor activities such as the defect rates, security findings, and production incidents (these should all improve). The goal is better security, not greater AI adoption.”
Untrusted, until proven otherwise
Arampatzis says DevSecOps teams need to treat AI-generated and AI-patched code as untrusted input until proven otherwise, applying the same standard as for any third-party contribution.
“That means provenance tracking, mandatory security testing before merge, and governance structures that can explain why a patch was accepted, not just that it passed a scan. Testing maturity stops being a checkbox and becomes the actual control layer protecting the pipeline.”
A great idea that most never implemented
Messdaghi says: “Everyone will tell you the answer is ‘shift left’, but let’s be honest, shift left was already a great idea that most organisations never actually implemented. Security got bolted onto the end of the pipeline in practice, no matter what the slide decks said. Layering AI-generated code on top of that unresolved problem doesn’t fix it. It just increases the volume flowing through a review process that was already under-resourced. So the real adaptation isn’t a new slogan. It’s finally building the review capacity and tooling that shift-left always assumed existed.”
She says to treat AI-written and AI-patched code as untrusted input, the same way you’d treat a new third-party dependency. “Track provenance so you actually know which commits were AI-touched. And test the patches themselves rather than trusting that ‘vulnerability closed’ means ‘risk closed’, because an AI fix can be syntactically correct and still introduce a new logic flaw nobody asked for.”
A look ahead
Looking ahead, we asked the experts what they see as the biggest cybersecurity challenge created by AI-generated and AI-patched code over the next three to five years.
Hibberd believes the biggest challenge will be establishing trust and provenance at scale. “We are trying to implement AI governance on top of data chaos. We need to get back to basics and fundamentals.
Organisations may soon operate software estates containing millions of lines of code that were generated, rewritten or patched by multiple AI systems. They may not know which model produced a particular component, what information the model relied upon or whether the person approving the change genuinely understood it.”
As ever, Hibberd says this is not a technology issue, it’s a business risk. “It’s only going to get worse if we don’t address this quickly.”
Building and maintaining trust
Moore says it’s about building and maintaining trust. “Customers don’t seem interested in who or what developed the app. They want: an easy-to-use site/app/platform; their data to be secure; their privacy to be respected; and the company to be upfront about where customer data goes and how it’s used. If a company can do marvelous things, but three years down the road customers learn, whether from reputation or, more likely, a data breach, that the company has deceived, hidden, and lied about the product or service, then trust goes out the window, and the customers may likely leave to find a newer company that has surfaced in those few years.
“Companies need to know that competitors are out there, and new ideas and approaches are arriving all the time, so keeping one’s reputation means taking the high road and playing the long game. AI can be good at gaining customers, but trust is vital to keeping them.”
The biggest challenge will be accountability
“Over the next three to five years, the biggest challenge will be accountability,” adds Malik. “When AI-generated or AI-patched code fails, organisations must know what changed, why it changed, who approved it and how to reverse it. The more we automate, the more important human judgement becomes. Security teams will need stronger verification, threat modelling, testing and rollback processes across DevSecOps.”
The scale of subtle failure
Messdaghi believes we’ll see the scale of subtle failure. “A human introduces one bad pattern at a time. AI can replicate the same flawed assumption across thousands of repos in an afternoon. The next major supply-chain-style incident probably won’t come from a single bad actor. It’ll come from a systemic blind spot baked into how models were trained or fine-tuned, surfacing everywhere at once. Detecting that kind of correlated risk is a very different problem than catching one vulnerable function.”
A false sense of security
In the end, Arampatzis says the biggest challenge over the next three to five years will not be a single catastrophic AI-written vulnerability. It will be a false sense of security: teams treating speed as a proxy for quality, confidence outrunning verification long before anyone notices the gap.
Information Security Buzz News Editor
Kirsten Doyle has been in the technology journalism and editing space for nearly 24 years, during which time she has developed a great love for all aspects of technology, as well as words themselves. Her experience spans B2B tech, with a lot of focus on cybersecurity, cloud, enterprise, digital transformation, and data centre. Her specialties are in news, thought leadership, features, white papers, and PR writing, and she is an experienced editor for both print and online publications.
The opinions expressed in this post belong to the individual contributors and do not necessarily reflect the views of Information Security Buzz.