Over 75% of organizations participating in McKinsey’s 2025 State of AI survey report using AI in at least one business function. However, concerns about AI-related risks still exist. The most common concern is the risk of inaccuracy or the inability to trust AI outputs.
Reviewing AI model outputs is crucial for increasing confidence, allowing you to use AI with confidence. However, the survey reveals there’s a significant variation in practices. At one end of the spectrum, 27% of organizations report reviewing all AI outputs. At the other end, 30% say few outputs are reviewed.
What’s interesting about AI is that we tend to focus on how it can replace people instead of remembering that it requires specific training to perform its tasks and is not a sole decision maker.
AI is like a summer intern who is learning the ropes. AI needs:
- time to understand your business
- coaching on how to interact in the work environment
- oversight of its output and actions
AI is not like a tenured employee you can fully rely on to consult the right resources, do their job well, and provide solid explanations that justify their actions or recommendations.
Once we internalize this, we can solve the AI trust problem.
A Vulnerability Management Use Case
Let’s examine an example of AI in cybersecurity, specifically how to confidently utilize AI within your vulnerability management program to reduce risk on network assets.
The vulnerability management process typically starts with the cybersecurity team running a vulnerability scan and then handing over the list of vulnerabilities on network devices to the network team to address. The network administrator needs to determine which vulnerabilities are relevant to the environment, how to remediate, and in what order.
There are three primary sources for trusted data about vulnerabilities: the National Vulnerability Database (NVD), the Cybersecurity and Infrastructure Security Agency (CISA) KEV catalog, and device manufacturer advisories. Since most organizations have devices from different manufacturers, network teams need to visit various vendor websites to pull information about the details of the vulnerability, affected device versions, the impact, workarounds, and when patches or updates will be available. Teams also need to consider the environment’s context and any active exploits targeting those vulnerabilities to determine the best actions to take.
This process requires extensive manual data aggregation and analysis, making it a compelling AI use case. AI can compile the relevant information, monitor for updates, summarize the issue, add context, and recommend courses of action. It does in minutes what would take hours, days, or even weeks to complete manually.
AI helps save time and scale, so you don’t miss critical vulnerabilities relevant to your environment. However, AI isn’t an expert in your field; you are. Mindlessly implementing recommended actions may have an adverse impact on the business.
Herein lies the solution to building trust in AI: Humans must validate the output.
Mitigating CVEs is not always straightforward. Often, several options are available depending on the mitigation actions available and what will work given the environment and resources.
- A patch may be available and recommended. However, a workaround may also be an option and preferable in the interim, so patching can happen at a time that is more acceptable to limit business impact.
- Or in lieu of a patch, a configuration change may be recommended to modify a service on a device that is critical to your network operations. Alternatively, the recommendation may be to disable a service that could disrupt important business operations.
As the expert, you need to understand the reason for the recommendation, validate it, and weigh the impact. Asking the model to show you the steps it took to arrive at this conclusion allows you to review the process, identify any potential missteps, and/or ask if there is an alternative.
With a clear explanation, you can determine the best course of action. You can ask AI to assemble the list of commands from device manufacturers’ web pages and technical documentation, so you have the steps to remediate. Again, you need to validate each step and adjust based on your knowledge and experience before making manual changes or choosing to automate. For example, there could be a common flaw. In network vendor A, the solution is to update to the next revision (say from 4.1.1 to 4.1.2) every single time. The model may infer that the steps are similar across vendors when, in fact, vendor B requires a full version upgrade (in this example, to version 5.x).
Trust but Verify
Just like it takes time to learn a foreign language, AI models need time to learn the language of your environment. After a few days of online language learning, you can string together some words that individually are correct. Someone who doesn’t know the language may be fooled, but to a native speaker, it’s gibberish. Mastering the language so that you speak like a native can take years of intensive training.
Similarly, challenges with hallucinations and bias will always exist, but can be lessened. The ‘trust but verify’ approach at multiple crucial points allows you to refine the process and improve the quality of the data provided to the model and its output. You can validate that the model is using trusted data sources to ensure there are no hallucinations. You can also prevent bias in the output by validating that the assumptions used in the decision-making process are not skewed in some way.
From Artificial to Authentic
Ultimately, AI may be artificial, but ensuring authenticity remains a human responsibility. With human-centered control, the model outputs improve over time, and your confidence builds. AI is not the sole decision maker; you will always need to verify, because you’re the expert. But you can save time, scale, and build accuracy and trust in the process.
Irfahn Khimji is a seasoned technology leader with a deep background in cybersecurity and go-to-market strategy. In his current role as Field CTO at BackBox, he leverages his extensive expertise to provide technical leadership and strategic direction. Recently, Irfahn served as Managing Director for Critical Start in Canada. His career progression highlights a strong focus on sales and leadership, including his tenure as Vice President at Tripwire, where he led sales, sales engineering, and channel teams for North America. He also served as the Managing Director for Qualys in Canada, overseeing technical sales, marketing, and services.
The opinions expressed in this post belong to the individual contributors and do not necessarily reflect the views of Information Security Buzz.