When OpenAI released ChatGPT to the public in November 2022, the world marvelled at the dawn of a new era in human-machine interaction. But as the broader public experimented with poetry and code snippets, another crowd watched quietly. They saw the potential for something darker.
Threat actors quickly realized large language models could be weaponized. The problem, is that ChatGPT, and others like it, came with guardrails. They wouldn’t write ransomware, wouldn’t craft phishing lures, wouldn’t help you breach a firewall.
Then came WormGPT.
First spotted in June 2023 on Hack Forums, WormGPT was a defanged version of ChatGPT, censorship removed, ethics stripped out. Tailored for cybercrime. Designed for those who wanted help writing malicious code or social engineering emails. For two months, it flourished. Then it vanished.
By August 2023, after journalist Brian Krebs unmasked one of its creators, WormGPT went dark. Its sudden exit was a clear signal. The developers, facing too much exposure, chose silence over prosecution.
But the silence didn’t last.
In the months that followed, “WormGPT” became less of a tool and more of a brand. On BreachForums (another gathering place for the digital underworld) new variants began to surface. This time, powered by stronger models. The badge remained the same, but the architecture had changed.
The New Worms
In October 2024, a BreachForums user calling himself xzin0vich unveiled a new WormGPT variant. It came with a Telegram chatbot interface, about 7,500 subscribers, and a clear mission: enable malicious automation. In February 2025, another variant appeared, this time from a user going by keanu. Both tools were available via Telegram and used a subscription and one-time payment model, with pricing details kept deliberately vague.
Cato CTRL analysts infiltrated both instances.
Their goal was simple: determine whether these tools were new LLMs altogether or just wrappers around existing models. They used prompts, jailbreaks, and forensic digging. The answers were telling.
The keanu variant eventually revealed its foundation: xAI’s Grok. Through probing questions, the chatbot disclosed it used Grok’s API but had been wrapped in a system prompt designed to disable safety constraints. The goal wasn’t to build a new model from scratch. It was to corrupt a legitimate one.
After multiple interactions, researchers managed to extract the system prompt, twice. In the second version, keanu’s authors had added language meant to block users from retrieving the prompt again. But the underlying method remained: override the rules, subvert the safeguards, exploit the power.
The xzin0vich variant told a similar story, with different code under the hood. It disclosed that it was built on Mistral’s Mixtral, a high-performing open-weight model. Beyond simply admitting its base model, the chatbot began to expose Mixtral-specific features under pressure, details such as top_k_routers and Grouped-Query Attention mechanics.
In both cases, the AI tools responded without hesitation to prompts requesting phishing emails, credential-harvesting PowerShell scripts, or obfuscation techniques. Whatever restraint the original LLMs had was neutralized. The threat actors had found a way to speak directly through them, using custom prompts as masks.
The Business of Crime
The original WormGPT had operated on a subscription model, €60 to €100 per month, or €550 annually. Private setups cost as much as €5,000. Its successors follow the same playbook. The market for uncensored AI is growing. These variants are not proof-of-concept. They are businesses.
FraudGPT, EvilGPT, DarkGPT, and others have joined the race. Some are likely vaporware, empty shells used to scam the scammers. But others, like these new WormGPT variants, are fully functional. Capable. Scalable.
Cybercrime has adopted AI not just as a tool but as infrastructure.
Behind the Screens
Cato CTRL’s findings show that attackers no longer need to build models from scratch. They can simply hijack existing ones. This drastically lowers the technical barrier to entry.
A system prompt here, a wrapper there, and a state-of-the-art model becomes a criminal assistant.
It’s a shift from writing code to writing instructions. From building software to crafting personas. These wrappers convince the LLMs to play a role, an attacker, an advisor, a forger. The sophistication lies not in the code, but in its manipulation.
The work of keanu and xzin0vich shows that today’s threat actors are increasingly literate in prompt engineering, system override methods, and LLM mechanics. They understand how to weaponize AI without breaking it. How to coax it, not just code it.
What Now?
For defenders, the challenge is layered.
First, there’s detection. Tools like Cato XDR, with behavioral analytics and user and entity behavior analytics (UEBA), are needed to spot anomalies. These models don’t live in a corporate environment, but their output does, through emails, scripts, or poisoned data pipelines.
Second, access must be tightened. Zero trust becomes essential. Stronger MFA, role-based access, and device posture checks can limit the spread when something slips through.
Third, education must evolve. Enterprises should simulate AI-generated phishing campaigns, monitor the use of GenAI in internal workflows, and update incident response playbooks to account for AI-assisted threats.
And finally, the enterprise must look inward. Tools like CASB dashboards can surface shadow AI usage, while secure development practices and CI/CD hardening help address risks posed by GenAI-generated code.
LLM Guadrails Are Not Perfect
Margaret Cunningham, Director, Security & AI Strategy at Darktrace says the proliferation of WormGPT variants underscores a growing reality in the AI threat landscape, LLM guardrails are not perfect. “As we’ve seen with WormGPT and similar findings like HiddenLayer’s universal jailbreak technique, threat actors will continue to find creative ways to skirt safeguards, expose system prompts, and remove censorship.”
She says we are already seeing the emergence of a jailbreak-as-a-service market, which could have wide-reaching implications. “These subscription models significantly lower the barrier to entry for threat actors, allowing them to leverage these tools without needing the technical skills to develop them themselves.”
Cunningham says it is important for organizations developing or deploying AI models to understand that guardrails do not guarantee protection against misuse. They are more like speed bumps. They can slow a threat actor down, but might not be enough to stop them. “Many organizations are still unprepared to defend against these threats. Static detections and rules won’t cut it. What’s needed is a more dynamic approach that can identify novel patterns and unexpected use of legitimate tools.”
A Catch-All Term
“Hundreds of uncensored LLMs exist in Dark Web communities,” says Dave Tyson, Chief Intelligence Officer at Apollo Information Systems. “Many of them are labeled “WormGPT” as a means of convenience, just like Americans say Kleenex for a facial tissue, even though the first is actually a brand versus the true item, a tissue. Some of them have distinctive names, like EvilGPT, but most criminal AI are glommed under the word “WormGPT” as a catch all term. Given the source code for WormGPT “leaked” and spread widely, it’s not especially surprising to see its dominance as a term or as backend code.”
Still, in actual cybercriminal operations, the vast majority of criminal activity leverages a chat channel or communication platform to “ferry” queries, he adds. “That creates a barrier of isolation between the AI and the actual user; It allows a criminal to provide a service (SaaS) to customers, but behind the scenes use any variety of models to meet the request.
Behind the scenes criminal can leverage models such as gemma or “small deepseek” models such as qwen, or llama.”
Pinpointing the Right Way to Attack
Tyson says local model servers, like LMStudio, can be used to run a choice of model, uncensored or jailbroken, via queries. “Use of chained AI prompts, such as tunneling WormGPT queries via FlowGPT can be used to set up a persuasive or social construct to perform criminal activity. This is a form of jailbreaking, of course. Speaking of jailbreaking AI, the core to this technique is getting the victimized AI to break its boundaries. Some of the simplest and most observed means to do this is by using a construct of historical research to hide nefarious activity; using the right paraphrasing to social engineer AI; or just leveraging an exploit of it.”
All of this discussion misses the use of the models, Tyson emphasizes. “Criminals are accelerating understanding and targeting, getting them faster to the decision to attack and pinpointing the right way to attack. That means cyberattacks can be more targeted or unique in execution. This drives increased return on investment through higher execution success rates.”
Bent to the Actor’s Use
“On the offensive security side of things, we talk about AI as a tool, a target, and a threat,” adds Trey Ford, Chief Information Security Officer at Bugcrowd. “From a platform provider perspective, this is an abuse of the intended use of these technologies, using GPT and GenAI as a threat (or a tool for attackers). We should fully expect GenAI to be abused in this manner, the use of local and open-weight LLMs in a jailbroken manner, where prompts and outputs are effectively jailbroken, on the regular.”
Ford says the use of any technology will be bent to the actor’s use (whether a paintbrush, spray can, or screwdriver) the skill of application is what is important. “My default frame is asking how to use the LLM as a toil-reduction tool. Yes, the technology is used to ‘help’ the threat actor – but it’s saving them time, not fully acting on their behalf. Yet. Ultimately the jailbroken or abused GPT isn’t the malicious actor, it’s the miscreant using and abusing the tooling.”
Information Security Buzz News Editor
Kirsten Doyle has been in the technology journalism and editing space for nearly 24 years, during which time she has developed a great love for all aspects of technology, as well as words themselves. Her experience spans B2B tech, with a lot of focus on cybersecurity, cloud, enterprise, digital transformation, and data centre. Her specialties are in news, thought leadership, features, white papers, and PR writing, and she is an experienced editor for both print and online publications.
The opinions expressed in this post belong to the individual contributors and do not necessarily reflect the views of Information Security Buzz.