Towards the end of March, OpenAI debuted image generation features for its ChatGPT-4o and ChatGPT-4o mini models. Less than a week later, the tool was made available for free to all users, and since then, users have reported that the feature can be used to create convincing fake documents—including receipts and passports.
According to the 2025 Cato CTRL Threat Report, generative AI (GenAI) tools like ChatGPT are lowering the barrier to entry for malicious actors.
The report highlights the growing threat of so-called “zero-knowledge threat actors”—people with no technical expertise who are now able to carry out advanced fraud using AI.
Cato CTRL researchers were able to generate a realistic fake passport using ChatGPT’s image generator. No jailbreaking or advanced tools were needed—just a few text prompts.
How Passport Forgery Has Changed
Back in the day, creating fake passports required specialized tools, skills, and access to underground networks. Forgers used software like Photoshop and distributed their work via dark web marketplaces—a process that could take days.
With AI image generation, that has changed. What once required skill and access has been whittled down to a matter of minutes and takes minimal effort.
ChatGPT’s image generator, originally designed for creative uses like producing avatars or illustrations, can be manipulated to make fraudulent documents.
In one instance, researchers uploaded a scanned passport and asked the tool to make changes. While the platform initially rejected the request due to policy restrictions, reframing the request as a “business card styled like a passport” bypassed those limits.
The tool then successfully altered names, photos, and other details to produce a realistic-looking passport.
The Rise of Zero-Knowledge Threat Actors
The report highlights that this ease of use gives rise to a new type of fraudster: someone with no prior experience in cybercrime or image editing. With just a few prompts, these “zero-knowledge” actors can now generate fake IDs and other documents that can be used in a wide range of fraud scenarios, including:
- New account fraud: Creating bank or credit card accounts using fake identities
- Account takeover: Calling companies to take over existing accounts, such as through SIM swapping
- Medical fraud: Altering prescriptions or insurance claims
- Financial scams: Modifying pay stubs or legal documents to apply for loans or manipulate contracts
The threat lies not only in how easy these documents are to create, but how convincing they’ve become. AI-generated images can now imitate details like handwriting, stamp textures, and official seals—details that used to be difficult to replicate without skill.
What Can Be Done?
The findings suggest a need for updated fraud detection strategies, using more than traditional tools that focus on phishing and malware. Instead, entities should consider more advanced detection methods to identify forged documents and suspicious use of generative AI tools.
The report ends with a caveat: as GenAI continues to improve, the quality of forgeries will likely increase.
That means it’s not just a technology issue—it’s a human one. Education, layered verification, and proactive fraud prevention measures will be key.
The Risks of Using AI-generated Code
Raj Kesarapalli, Director of Product Management at Black Duck, says GenAI is increasingly used to write code and is improving rapidly.
While GenAI offers a great starting point for developers, he says it is still important to validate the following:
- The generated code addresses the intended functionality and nothing more
- The generated code quality is good and addresses the error conditions and edge cases
- The generated code is optimized for performance
- The generated code doesn’t have any security vulnerabilities
“LLMs will be used to create a significant percentage of mundane portions of the code,” adds Kesarapalli. “For the last two decades, even before GenAI came into the picture, developers have become very comfortable copying/pasting code from sites such as Google, StackExchange, GitHub, etc. LLMs take this to the next level. It is important to note that the generated (or copy/pasted code) gives developers a starting point so they don’t have to code from scratch. Once the initial snippet of code is generated, developers must extend, integrate, and test this generated code to fit their needs.”
Prioritizing Convenience Over Security
He says most developers don’t completely understand the ramifications of the risks involved with using AI-generated code as they are focused on delivering functionality most conveniently or easily. “Even before AI-generated code came into the picture, developers were augmented with tools and processes that addressed functional testing, code quality, security vulnerabilities, and performance bottlenecks.
Kesarapalli says these validation/verification steps are even more critical given the rise of Gen AI. “In addition to the existing concerns, enterprises must now ensure that LLMs are not injecting malicious code that won’t be caught with existing tools and processes.”
Hallucinations, along with intentional malicious code injection, are definitely a concern. Hallucinations result in unintended functionality whereas malicious code injection results in security concerns. It’s important to use a certified LLM trained on trusted code and to review the code generated and inserted into the code base to manage software risk. Additionally, it’s essential to identify AI-generated code in the codebase so that the code can be identified and evaluated. Peer reviewers should also be aware of the generated portion of new code so that they can review it in a different light.
The Risks of Slopsquatting
The use of LLMs to at minimum assist in development has proliferated in a massive way, says Casey Ellis, Founder at Bugcrowd. “The key risks highlighted by slopsquatting are over-reliance on the trustworthiness of LLM output, and the increase in development velocity not being met by an increase in assurance.”
While unique to AI-generated code, the “slopsquatting” issue follows a familiar pattern whenever new, competitive, speed, and/or quality-delivering technological advances are introduced: Haste is the natural enemy of quality, and security is quality’s child, adds Ellis.
In ending, Ellis says it’s important to remember that developers are motivated to “make the thing work” as opposed to “making sure the thing doesn’t do all of the things it potentially shouldn’t.” When this misalignment exists, issues like this exist, and should an accelerating function like AI-generated code be added, attacks like slopsquatting are the natural byproduct.
Information Security Buzz News Editor
Kirsten Doyle has been in the technology journalism and editing space for nearly 24 years, during which time she has developed a great love for all aspects of technology, as well as words themselves. Her experience spans B2B tech, with a lot of focus on cybersecurity, cloud, enterprise, digital transformation, and data centre. Her specialties are in news, thought leadership, features, white papers, and PR writing, and she is an experienced editor for both print and online publications.
The opinions expressed in this post belong to the individual contributors and do not necessarily reflect the views of Information Security Buzz.